Important: notes get deleted when they contain certain characters in the title

[florianklumb]

Actual behaviour

Put a question mark into a note title and baaam: it gets deleted.

General Info

Current Date: Fr Aug 18 14:14:18 2017
Version: 17.08.6
Build date: Aug 17 2017
Build number: 3180
Platform: windows
Operating System: Windows 7 SP 1
Build architecture: i386
Current architecture: x86_64
Release: AppVeyor
Qt Version (build): 5.7.1
Qt Version (runtime): 5.7.1
Portable mode: no
Settings path / key: \HKEY_CURRENT_USER\Software\PBE\QOwnNotes
Application database path: C:\Users\LENOVO\AppData\Roaming\PBE\QOwnNotes\QOwnNotes.sqlite
Application arguments: E:\Programs\PortableApps\QOwnNotes\QOwnNotes.exe
Qt Debug: 0
Locale (system): de_DE
Locale (interface): en
Icon theme: breeze-qownnotes
Notes in current note folder: 116
Enabled scripts: 0

[pbek]

Yes, ? does not work on many file systems, so we'll remove that...

[florianklumb]

Ehm ... Patrizio, you cannot be serious with closing this (in which case, I will strongly consider stopping to use QOwnNotes). It is not the responsibility of the user to make sure, that file names are correct. But even if you wanted to delegate that responsibility to the user (and show an error message) that would be a 1000 times better than DELETING the note without the user knowing about it. This is a prio A bug.

[pbek]

I can't reproduce any deleted notes. For me just the ? just gets deleted from the filename.
So I thought your were talking about that a part of the note title (or filename) gets deleted.

Can you please explain in more detail and step by step what you did, what happened and what you expected. And is there any useful log output if you enable the log widget?

[florianklumb]

It's the same as in #683:

[Aug 18 15:10:01] [critical]: "Could not store note file: .../Notes/new note with question mark ?.md - Error message: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch."

Reproduction is as easy as creating a note and adding a question mark to the first line.

If I click "Show log" nothing happens. Maybe that is another bug?

[pbek]

If I click "Show log" nothing happens. Maybe that is another bug?

It's just a shortcut to enable the log panel, maybe you have resized that panel very small?

[pbek]

I never had the problem your are describing on my Windows test installation, but I can try to reproduce it.
On which filesystem are you storing your notes? That's a thing QOwnNotes will never know and there are quite a lot of filesystems with their own restrictions... People could use 10 different filesystems, mounted into the same note folder. The app wouldn't see this...

PS as a reminder: This is a one-person spare-time project with no funding. 😸

[pbek]

PPS: If you want to help fix prio A bugs... reasonable pull requests are always welcome!

[florianklumb]

I know, Patrizio. I think you're doing an awesome job - I appreciate your efforts as much as I appreciate every open-source project.

I am not aware of resizing the log panel but also not sure where to find.

IMHO there is a case to make for always parsing file names and eliminating illegal characters of supported file systems. I mean QOwnNotes is a plain file note taking app but the focus is - again IMHO - on the files being perfectly named.

I am sorry but I am afraid that I cannot help with development.

[pbek]

I know, Patrizio. I think you're doing an awesome job - I appreciate your efforts as much as I appreciate every open-source project.

:) thank you

I am not aware of resizing the log panel but also not sure where to find.

it's usually somewhere on the bottom, but you want to take a look in the Windows settings if it's turned on

IMHO there is a case to make for always parsing file names and eliminating illegal characters of supported file systems.

and there lies the crux, QOwnNotes doesn't know the file-systems users use...

[florianklumb]

You might not exactly know what fs is being used but you can make an educated guess what is likely to be used. You know people use FAT and NTFS and so you forbid all characters which are illegal in filenames on those fs, see also here: https://kb.acronis.com/content/39790

My take on this would be - and I am repeating what I have said before: the focus of QOwnNotes IMHO is
NOT on the files being perfectly named. So just replace all special characters in filenames with underscores. But if you do not want to do that at least check for those special characters and inform the user. And, on top of that, catching critical errors like the one from above while saving notes would be wise I think.

[fomoshlomo]

Hi, I just ran into this myself a couple of days ago inadvertently and unintentionally - I added a question mark to a note title for something that I wasn't sure about and was surprised when the note disappeared. Luckily, I'd only copied and pasted a few lines from another program into the note, so I didn't lose anything permanently.

As a user of multiple operating systems, I'd like for notes to be saved with filenames that work on any platform. Besides that, if I forget or make a mistake and include a character in the note title that the file system wouldn't like, I don't want it to become a critical and irrecoverable error.

I did a little testing - details below - and I'm happy to try and fix this by either 1) removing these characters :\/<>"|?* from filenames, or 2) as @yodayeti suggests, converting them to another character, like underscores. This should also fix #683.

I don't know whether there may be other characters that are disallowed in filenames on other platforms, but I can look around a little.

Edit: After some brief Google searches, I didn't see an "official" article from Apple on the subject, but it seems that only :, /, and \0 could cause problems on macOS [1][2][3][4].

I feel like ideally, the note list would display the actual note title instead of the now-mangled filename, but maybe that should be a separate issue/request.

[1] http://www.informit.com/articles/article.aspx?p=1144082&seqNum=5
[2] https://stackoverflow.com/questions/20914649/what-characters-are-illegal-on-the-filename-on-ios-or-os-x
[3] https://ss64.com/osx/syntax-filenames.html
[4] https://discussions.apple.com/thread/1430177?tstart=0

---

Results of testing

I tested saving notes containing characters that are considered reserved on Windows.

On Windows

These characters are OK - they just get removed from the filename: :\/

For these characters: <>"|?*

• When creating a new note with one of these characters, the note disappears from the note list immediately and no file appears to be created.
• Sometimes (but not always), when changing the name of an existing, saved note to a new name that contains one of these characters, the file gets deleted and/or an entry for the note stays in the note list until QOwnNotes is restarted. When an entry for the note stays in the note list and I select something else and then come back to the badly named note, the note is not editable. This seems to be the same problem referenced in Not being able to edit notes with a " in the title #683.

On Linux

None of these characters presented a problem for me; :\/ were removed from the filename and <>"|?* stayed in the filename.

General Info

Current Date: Fri Aug 18 21:28:59 2017
Version: 17.08.6
Build date: Aug 17 2017
Build number: 3180
Platform: windows
Operating System: Windows 10
Build architecture: i386
Current architecture: x86_64
Release: AppVeyor
Qt Version (build): 5.7.1
Qt Version (runtime): 5.7.1
Portable mode: yes
Settings path / key: D:/software/QOwnNotes/Data/PBE/QOwnNotes.ini
Application database path: D:\software\QOwnNotes\Data\QOwnNotes.sqlite
Application arguments: QOwnNotes.exe, --portable
Qt Debug: 0
Locale (system): en_US
Locale (interface): empty
Icon theme: breeze-qownnotes
Notes in current note folder: 43
Enabled scripts: 0

[pbek]

The problem is if I just remove or replace those characters it will mangle the filenames for all other users that don't use file systems with those restrictions and it's even worse if people are using the same notes on multiple computers with different file systems... it it is really getting messy if your throw Nextcloud notes (who also uses the headline as filename) into the game then notes are constantly removed in a ping pong fashion...

[fomoshlomo]

With the current behaviour, a user who uses multiple OSes could create a note on an OS with less restrictions on filenames and be surprised when they open QOwnNotes on Windows and discover that a note is missing.

As for file syncing software - here's what I found:

• The ownCloud documentation states that it is "up to the interface software to replace the not allowed characters." If I understand this correctly, this means that the responsibility is on QOwnNotes.
• Nextcloud is supposed to let the client [software?] or user know that files could not be synced.

I don't use ownCloud or Nextcloud, but they shouldn't care about what's in the note or how QOwnNotes finds the title of the note, should they? Wouldn't they only care about the filenames that QOwnNotes has chosen to sync? Not that QOwnNotes should, but QOwnNotes could assign a random, unique string of characters (e.g. "a2tdW6x") to notes and map note titles to those random filenames in a database - and ownCloud/Nextcloud wouldn't care.

I feel like no matter which OS is used, QOwnNotes should save (and look for) filenames in the same way: if it replaces characters on one OS, it should replace characters on all of them. If it always replaces the 9 potentially problematic characters, this will avoid the "ping pong", because there will never be a problem syncing the files on different file systems.

In the end, though, either 1) mangling filenames or 2) saying "Hey, I couldn't save that note because of a problematic note title" (on Windows) or "Hey, there's no problem here, but you might want to use a different note title, because you won't be able to access this note on Windows" (on non-Windows platforms) would be preferable to allowing users to shoot themselves in the foot.

[pbek]

Not that QOwnNotes should, but QOwnNotes could assign a random, unique string of characters (e.g. "a2tdW6x") to notes and map note titles to those random filenames in a database

There is no persistent note database in QOwnNotes.

I feel like no matter which OS is used, QOwnNotes should save (and look for) filenames in the same way: if it replaces characters on one OS, it should replace characters on all of them.

Almost all filesystems (it seems aside from FAT and NTFS) have no problems with those characters and their note file names would get changed which will cause distress for the majority of users...

saying "Hey, I couldn't save that note because of a problematic note title" (on Windows)

notes are stored in the background, I've yet to find a way to get around this

or "Hey, there's no problem here, but you might want to use a different note title, because you won't be able to access this note on Windows" (on non-Windows platforms) would be preferable to allowing users to shoot themselves in the foot.

why would the macOS and Linux users care? :)

[fomoshlomo]

There is no persistent note database in QOwnNotes.

Ah - for some reason, I thought there was; but no matter.

note file names would get changed which will cause distress for the majority of users...

Yes, disrupting things for existing users would be the biggest argument against this change. (However, I think this would affect very few people - it will only affect non-Windows users who have saved notes with these problematic characters in the titles. The effect would be that if they start looking at the note files directly, the files would have different filenames; this could cause a problem if, for whatever reason, another program or a shortcut depended on the QOwnNotes note file.)

why would the macOS and Linux users care? :)

Those of us who love QOwnNotes so much that we use it on multiple platforms would care! 😃

I admit that this problem isn't necessarily a big one or a huge concern - if other users are like me, they rarely (if ever) feel the need to use any of :\/<>"|?* in note titles. The part that concerns me is that currently, on Windows, including one of those characters in a note title is an irrecoverable error.

[pbek]

Phew, it took three hours to get that in...

17.08.7

• old notes that were automatically renamed will now only be removed if the new
  note was stored successfully
• if a note gets renamed automatically because the note title changed it will
  now be checked if the note can actually be stored to the note folder before
  really storing it to be able to attempt to replace additional characters in
  the filename before storing it again in case the note is stored on a
  filesystem with harsher filename restrictions
  • if the note still cannot be stored to disk the 2nd time the error will be
    logged to the log panel like it already was before

[pbek]

There now is a new release, could you please test it and report if it works for you?

[fomoshlomo]

Tested 17.08.7 successfully on Windows and Linux. On Windows, characters that are not allowed in filenames get converted to underscores; on Linux, the filenames stay as they are - thanks!

[pbek]

Great, thank you for testing!

[florianklumb]

Great work, Patrizio! Thanks a bunch!

First bullet point - not sure how to test.

Second bullet point test:

• Illegal chars in file names get replaced and notes are not deleted anymore on Windows - okay!
• Notes with replaced file names like a) "is this true?" => b) "is this true_" show up with modified title (as b) in the note overview - not okay

[pbek]

Great work, Patrizio! Thanks a bunch!

It was some intense work!

Notes with replaced file names like a) "is this true?" => b) "is this true_" show up with modified title (as b) in the note overview - not okay

Are you talking about the note list?
Filenames are shown there, so you will see the new filename in that list.

[florianklumb]

You could also show the title as the title and the file name are always in sync. That would mitigate actually the problem of character replacement completely from an in app perspective.

[pbek]

I'm not sure what you are talking about, can you please explain in more detail.
You can also visit the General settings and take a look at the Note settings, maybe you'll find some gems there. ;)

[florianklumb]

• I create a note called "note?", it is saved now as "note_"
• First line of the note and the file name are always synchronized
• So: why not show the first line in the note overview instead of the file name?

[pbek]

You have still not explained what the note overview is for you... :)
You can turn off that "synchronization" and then rename the filename in the note list...
But apart from that you could make sense, although it would be not very easy to implement.
Make a new feature request and I'll think about it.

[florianklumb]

I guess you call it "Note list panel".