Add inking support to calculator for basic calculations/conversions or other math needs

[arcadiogarcia]

Problem Statement

The calculator app currently supports different input mechanisms (keyboard, mouse, touch, pen), and while the app interface has been tailored in some way for for the rest of them, pen input has received less attention: the pen is treated like a regular mouse pointer with no special consideration. This is less than ideal when trying to input long numbers in a device where pen is one of the primaty input methods (e.g. Surface Studio in canvas mode, Surface Go with no keyboard attached), where many users would find easier to just write down the numbers.

Evidence or User Insights

Many Windows surfaces have benefited lately from tighter ink integration, from the new inline text input canvas to the integration of ink in apps like the Mail client, the Office suite and Microsoft To-Do. Users of pen enabled devices have come to expect more intuitive ways to interact with their computer, with the pen allowing rich ink input rather than just behaving like a simple pointer. The community response to this suggestion has been very positive suggesting that at least high-confidence users would find it a welcome addition (no pun intended).

Proposal

The proposal is to allow user to enter calculations in handwriten form, which will be recognized, parsed and calculated as any other user input.

Goals

• Allow active pen (e.g. Surface Pen) users to enter handwritten calculations, recognize them and display the result
• Understand mathematical expressions that can be expressed in one line using the same limited expression format the calculator already recognizes for pasted strings. E.g.:
  • VALID: 1+ 2 * 3
  • NOT VALID: sin(1)

Non-goals

• Provide handwriten input for touch users
  • Accuracy wouldn't be satisfactory and it would conflict with the current touch usage patterns
• Provide feature parity with the Math Input Panel
  • It would be a nice follow-up but it would require additional work on the recognition logic
• Replace the current numeric pad. It should remain available so the user is free to use the pen as a pointer if they prefer it.

Low-Fidelity Concept:

I was playing around with the code and I added support for entering expressions using ink:

It works by overlaying a InkCanvas on top of the calculation result, and leveraging the existing OnPaste method to process the recognized string.

Is this a feature you would be interested in adding? If you do I can polish it and send a PR 😊

Note: A team member edited this comment for clarity based on conversation below.

[danbelcher-MSFT]

Arcadio, this is great! We are still in the process of transitioning into the open, so I don't think anyone was expecting contributions so soon. Adding ink support into Calculator is definitely on our feature backlog. I think the next step here is to bring our PM into the conversation so we can refine our vision for how this feature should appear in the app.

[HowardWolosky]

Indeed, thanks so much, Arcadio! This looks very cool (and useful). I know we did some investigation into pen-based input a couple times in the past, along with (I think) some user research. We're going to find the work that was previously done in this area to help inform us of how to best move forward with you on this.

[MicrosoftIssueBot]

This is your friendly Microsoft Issue Bot. I've seen this issue come in and have gone to tell a human about it.

[yaira2]

This is so cool, I would love this feature.

[pengzh0928]

Love this feature.

[newtykip]

This would be great!

[lizhimins]

I really hope that it can support some complex operation rules.

If only some simple operations are supported, because the accuracy of keyboard input is far greater than that of recognition, users will choose keyboard input. Handwritten recognition is required only when the user is not easy to input the keyboard content. However, recognition of complex rules is still not good enough.

[arcadiogarcia]

@lizhimins My proposal here would be to leverage the preexisting logic for pasting operations. If by complex rules you are thinking about things like fractions and roots that would require additional work to make the ink to text parser understand them since that is not provided by the UWP APIs. My intent here wasn't to make keyboard users switch to pen input, just provide a good alternative for keyboardless devices or people that prefer using the pen.

[grochocki]

@arcadiogarcia When you get the chance, can you update your idea with more details following the feature pitch template from our New Feedback Process?

[arcadiogarcia]

Sure! This would be the feature pitch:

Problem Statement

The calculator app currently supports different input mechanisms (keyboard, mouse, touch, pen), and while the app interface has been tailored in some way for for the rest of them, pen input has received less attention: the pen is treated like a regular mouse pointer with no special consideration. This is less than ideal when trying to input long numbers in a device where pen is one of the primaty input methods (e.g. Surface Studio in canvas mode, Surface Go with no keyboard attached), where many users would find easier to just write down the numbers.

Evidence or User Insights

Many Windows surfaces have benefited lately from tighter ink integration, from the new inline text input canvas to the integration of ink in apps like the Mail client, the Office suite and Microsoft To-Do. Users of pen enabled devices have come to expect more intuitive ways to interact with their computer, with the pen allowing rich ink input rather than just behaving like a simple pointer. The community response to this suggestion has been very positive suggesting that at least high-confidence users would find it a welcome addition (no pun intended).

Proposal

The proposal is to allow user to enter calculations in handwriten form, which will be recognized, parsed and calculated as any other user input.

Goals

• Allow active pen (e.g. Surface Pen) users to enter handwritten calculations, recognize them and display the result
• Understand mathematical expressions that can be expressed in one line using the same limited expression format the calculator already recognizes for pasted strings. E.g.:
  • VALID: 1+ 2 * 3
  • NOT VALID: sin(1)

Non-goals

• Provide handwriten input for touch users
  • Accuracy wouldn't be satisfactory and it would conflict with the current touch usage patterns
• Provide feature parity with the Math Input Panel
  • It would be a nice follow-up but it would require additional work on the recognition logic
• Replace the current numeric pad. It should remain available so the user is free to use the pen as a pointer if they prefer it.

Low-Fidelity Concept:

See the .gif above 😊

[grochocki]

This is great, thanks! I am going to update your original comment with these new details.

[ghost]

We reviewed the pitch and would love to explore this idea further! The pitch is a great start, but there are still some open questions. Moving this into planning to iron out some of those details. A human will follow up with some feedback on your pitch shortly. Keep in mind that not all ideas that make it into the planning phase are guaranteed to make it to release. For more information on next steps, check out our spec workflow.

[grochocki]

@arcadiogarcia We love the idea of adding inking support to Calculator, and I think your sample is a great proof-of-concept! We have prototyped some inking integrations in the past, but we ran into a couple roadblocks mostly around error detection/correction. I'll try to dig up some more details on what we have tried.

Before submitting a PR, we'll want to work through some of those open issues as well as figure out the right UX approach (e.g., should any calculator input support ink? should it be a separate mode? etc.). Also, while inking can be a really cool way to interact with an app, we would like to think through some of the user scenarios where a user chooses ink over other input methods, so we can ensure we are optimizing for those scenarios.

I created calculator-specs/ink to track progress.

[arcadiogarcia]

Great! What is the best place to discuss those open issues, this thread or opening a PR with a draft of the spec and iterating there?

[mdtauk]

Would this not be covered by using the Handwriting TextBox control?

https://docs.microsoft.com/en-us/windows/uwp/design/controls-and-patterns/text-handwriting-view

[arcadiogarcia]

Currently the Calculator app doesn't launch the Handwriting TextBox control when tapped with a pen. An alternative approach would indeed be to just pop up the handwriting control, but it is optimized for text input and it doesn't support adding our own recognition logic. With the raw InkCanvas approach we can easily filter the candidates that aren't valid arithmetic expressions to improve accuracy, while I believe the Handwriting view doesn't give us a list of the candidates, only the closest match.

[grochocki]

Great! What is the best place to discuss those open issues, this thread or opening a PR with a draft of the spec and iterating there?

Ultimately, everything should be documented in the spec, so it would be good to have a draft spec drafted to start from. If the open questions are blocking that from being created, then we should have those conversations here first.

[grochocki]

Here are a couple of the larger open questions we had when we prototyped inking support in Calculator in the past:

Is inking a separate mode or is inking pervasive and enabled when you touch you pen to calculator?

We explored a couple options, including (a) having an independent "inking" calculator mode, (b) having the input field respond to pen input (as you have in your prototype), or (c) a hybrid that detects when you are interacting with Calculator with a pen and puts you in a more immersive math ink work space. They each have pros and cons, but IMO, but (b) or (c) would feel much more natural for pen users. Here is one early sketch more along the lines of (a)/(c):

How do you handle error detection/correction for invalid input?

This question mostly speaks for itself, but when entering a complex equation, it would be unfortunate to just drop the input and have the user enter it all again. We would need to be able to detect and allow for error correction for invalid input. Also, how accurately could we detect some functionality? For example, if someone wants to input cubic root of 27, will we recognize that properly? Or would we detect it as 3 * square root of 27? Here is one design exploration:

We may be limited to operations from Standard mode. How do we convey what you can/can’t to the user?

It is unclear which math expressions we could support. For example, I believe Standard mode operations should work, but it is not as clear to whether all of the functionality of Scientific or Programmer modes could be supported. If some functionality is limited, how do we convey that to the user? If we went with approach (b) from above, how do we handle the user of parenthesis in Standard calculator where parenthesis are not supported?

[jingkecn]

Maybe you would be interested in this: https://www.myscript.com/interactive-ink/ (it would be nice if we could integrate interactive-ink SDK in the calculator)

[Ashesh3]

This is a really helpful feature for users like me who primarily work with their pen. What is the status on this issue?