Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unicode support. #8

Open
prataprc opened this issue May 16, 2017 · 2 comments
Open

Unicode support. #8

prataprc opened this issue May 16, 2017 · 2 comments
Assignees
@prataprc
Labels
bug

Comments

@prataprc
Copy link
Owner

Default Scanner (SimpleScanner) and Terminal parser functions should have unicode support, along with utf8 encoding.
@prataprc prataprc added the bug label May 16, 2017
@prataprc
Copy link
Owner Author

prataprc commented Jun 7, 2017

While at it, add support for white spaces in unicode, for instance SimpleScanner.SkipWS().

@prataprc prataprc self-assigned this Jun 29, 2017
prataprc added a commit that referenced this issue Jul 3, 2017
@prataprc
Added test case for unicode parsing #8.
8589a49
prataprc added a commit that referenced this issue Jul 3, 2017
@prataprc
Draft implementation for checking unicode whitespace #8.
5385952
@prataprc
Copy link
Owner Author

prataprc commented Jul 3, 2017

Skipping Unicode whitespace can be achieved by using SetWSPattern() API.
By setting the necessary whitespace unicode-points.

There is also unicode category called, Separator, Spaces
http://www.fileformat.info/info/unicode/category/Zs/list.htm

Can be used with golang's regexp \p{}: https://golang.org/pkg/regexp/syntax/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prataprc prataprc

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@prataprc