Provide a locationHelper module for code related to locations, rectangles and points #7537

LeonarddeR · 2017-08-29T07:08:35Z

Link to issue number:

Summary of the issue:

Currently, there are many different ways in which a location is presented in NVDA:

(left, top, width, height)
(left, top, right, bottom)
textInfos.Rect, which has left, top, right and bottom properties
ctypes.wintypes.Rect, which is a struct and also has left, top, right and bottom properties
(x,y)
textInfos.Point, which has x and y properties
ctypes.wintypes.Point, which is a structure and also has x and y properties

On the fly conversion between the different types is not possible. Also, conversion from screen coordinates to client coordinates, physical to logical coordinates, etc. is somewhat cumbersome.

Implementation details

This pr adds a new locationHelper module to get rid of these inconsistencies and conversion code that must be duplicated several times. This includes new namedtuple based classes with some extra enhancements:

Location: a named tuple with the four properties above, including right and bottom properties which are automatically calculated. This takes away the need to convert locations to rectangles and back in most cases.
Rect: also a named tuple with corresponding properties, including width and height properties.
Both Location and Rect are based on a mix-in class which provides the following facilities
- .toCRect(): converts Location or Rect to a Ctypes rectangle
- .toLogical(hwnd): Converts Location or Rect to logical coordinates, assuming that the current coordinates are physical
- .toPhysical(hwnd): Converts Location or Rect to physical coordinates, assuming that the current coordinates are logical
- .toScreen(hwnd): Converts Location or Rect to screencoordinates, assuming that the current coordinates are client coordinates
- .toClient(hwnd): Converts Location or Rect to client coordinates, assuming that the current coordinates are screen coordinates
- property topLeft: returns a Point for the top left coordinates
- property bottomRight: returns a Point for the top left coordinates
- property center: returns a Point for the center coordinates
- Point(x=3,y=10) in location(left=2,top=2,width=20,height=20) will return True, etc. You can also check whether one Location or Rect is part of another Location or Rect. You can even do this for ctypes or TextInfos points and rectangles
- self<other: Returns whether self is a subset of other (i.e. whether all points in self are contained by other)
- self>other: Returns whether self is a superset of other (i.e. whether all points of other are contained by self)
- self-other: returns a new Location or Rect with substracted coordinates
- self & other: Returns the intersection of self and other. For example, if self = Rect(left=10,top=10,right=25,bottom=25) and other = Rect(left=20,top=20,right=35,bottom=35), this results in Rect(left=20,top=20,right=25,bottom=25)
Point: named tuple with x and y properties. It provides the following facilities:
- self+other: returns Point(self.x+other.x, self.y+other.y)
- self<other: Returns whether self is less than other. This first compares y, than x coordinates. For example: (x=4,y=3) < (x=3,y=4) because self.y is less than other.y. This makes it possible to sort objects based on their location on screen. To compare in opposite order (i.e. compare x, than y), use tuple(self) < tuple(other). This also applies to self<=other, self>other and self>=other
- self - other: returns Point(self.x-other.x, self.y-other.y)
- .toLogical, .toPhysical, .toScreen, .toClient
Helper functions (toRect, toLocation and toPoint) to convert between the new locationHelper classes, textInfos and ctypes ones.

The new module is currently only used in the following situations.

The location property for all core objects
displayModel.getWindowTextInRect

Known issues

One concern about changing the Point and Rect objects from textInfos into named tuples is that named tuples aren't mutaple. There are some places in NVDA Core where this would lead to issues, however they could easily be solved using the several helper methods and operator support. Regarding backwards compatibility with add-ons, it is best to keep textInfos.Point and textInfos.Rect for now. I've added a deprecated statement to the doc string of these classes.

@josephsl: as a prominent person in the add-ons community, you might be able to tell what add-ons cause compatibility issues if we convert all textInfos.Rect and textInfos.Point cases into locationhelper ones?

* Added the ClientToScreen function from user32.dll to the winUser module * Use locationHelper for object locations

dkager · 2017-09-23T11:18:22Z

source/locationHelper.py

+
+	def __add__(self,other):
+		if not isinstance(other,_POINT_CLASSES):
+			return NotImplemented


Why do you not throw a NotImplementedError?

See this on Stack Overflow

dkager · 2017-09-23T11:20:13Z

source/locationHelper.py

+class Point(namedtuple("Point",("x","y"))):
+	"""Represents a point on the screen."""
+
+	def __add__(self,other):


Maybe a quick line of PyDoc here to summarize what this function does.

dkager · 2017-09-23T11:20:35Z

source/locationHelper.py

+			return NotImplemented
+		return Point((self.x+other.x),(self.y+other.y))
+
+	def __radd__(self,other):


Maybe a quick line of PyDoc here to summarize what this function does.

I'm a bit reluctant to add PyDoc to every operator overloader, I'd say and would be enough?

So what does (r)add do in this case?

dkager · 2017-09-23T11:20:58Z

source/locationHelper.py

+	def __radd__(self,other):
+		return self.__add__(other)
+
+	def __lt__(self, other):


Now you suddenly put spaces after commas. :)

dkager · 2017-09-23T11:21:30Z

source/locationHelper.py

+	def __lt__(self, other):
+		"""
+		Returns whether self is less than other.
+		This first compares y, than x coordinates.


than --> then

dkager · 2017-09-23T11:43:54Z

source/locationHelper.py

+				return False
+		return other.left == self.left and other.top == self.top and other.right == self.right and other.bottom == self.bottom
+
+	def __neq__(self, other):


Here too I would just call __eq__.

dkager · 2017-09-23T11:45:24Z

source/locationHelper.py

+				return False
+		return not (other.left == self.left and other.top == self.top and other.right == self.right and other.bottom == self.bottom)
+
+	def __sub__(self,other):


Do you not define __add__?

No, since I have no solid idea about what add should do. Note that for sets, you can also only subtract them.

dkager · 2017-09-23T11:48:05Z

source/locationHelper.py

+		return Rect(self.left,self.top,self.right,self.bottom)
+
+class Rect(_RectMixin, namedtuple("Rect",("left","top","right","bottom"))):
+	"""Represents a rectangle on the screen.


Could you add a short sentence summarizing how this is different from a Location?

dkager · 2017-09-23T11:57:45Z

source/locationHelper.py

+	"""
+	Converts the given input to L{Rect}.
+	Input should be one of the following types:
+		* One of l{_RECT_CLASSES}.


Judging by the underscore, this is a private variable, which should probably not be in your documentation.

This part of the docs should definitely be public, so I'll also make these variables public.

dkager · 2017-09-23T11:58:05Z

source/locationHelper.py

+	Converts the given input to L{Rect}.
+	Input should be one of the following types:
+		* One of l{_RECT_CLASSES}.
+		* One of L{_POINT_CLASSES}: converted to L{Rect} square of one pixel.


Same, also for the functions below.

feerrenrut

I think this is a good direction to move in, there are a few concerns I have with this implementation though.
It's not very clear what the difference between Location and Rect is. To me Location sounds like a Point. I think Rect and Location could be named better to highlight their differences.

feerrenrut · 2018-01-03T01:40:48Z

source/locationHelper.py

+		return Point((self.x+other.x),(self.y+other.y))
+
+	def __radd__(self,other):
+		"""Returns a new L{Point} with x = self.x + other.x and y = self.y + other.y."""


Unit test:
myPoint = sum([point1, point2, point3])
I think this will give an error, you need to check for other==0 and then return self See http://www.marinamele.com/2014/04/modifying-add-method-of-python-class.html

feerrenrut · 2018-01-03T01:43:39Z

tests/unit/test_locationHelper.py

+#See the file COPYING for more details.
+#Copyright (C) 2017 NV Access Limited, Babbage B.V.
+
+"""Unit tests for the locationHelper module.


Thanks for adding unit tests, I have some experience with this kind of code through games and simulation development and can say that it is fraught with edge cases. I would like to see the tests expanded. I'll make notes through the code for edge cases I worry about, would you mind adding unit tests for these?

feerrenrut · 2018-01-03T01:44:45Z

source/locationHelper.py

+		return Point((self.x-other.x),(self.y-other.y))
+
+	def __rsub__(self,other):
+		return self.__sub__(other)


As with __radd__ please add a unit test.

Could you elaborate? For add and radd, we have the sum function, but I belief there is no such function for subtraction, right?

Hmm, I didn't think through how this might get called. Can you answer that, how would this get called? Why is it necessary? Is there some case (as with __rsum__) that it might get called with other as an integer?

This would get called if you do ctypes.wintypes.POINT(3,3) + locationHelper.Point(4,4), in which case it should return locationHelper.Point(7,7)
May be that's just a valid ctest case to work with? :)

Sorry, I didn't follow that. My questions were about the __rsub__ function. I found the following on the python docs

These functions are called to implement the binary arithmetic operations (+, -, *, /, %, divmod(), pow(), **, «, », &, ^, |) with reversed operands. These functions are only called if the left operand does not support the corresponding operation. For instance, to evaluate the expression x-y, where y is an instance of a class that has an __rsub__() method, y.__rsub__(x) is called. Note that ternary pow() will not try calling __rpow__() (the coercion rules would become too complicated).

So adding a test to show that:
ctypes.wintypes.POINT(3,3) - locationHelper.Point(1,1) is locationHelper.Point(2,2)
and also perhaps a test that shows that:
0 - locationHelper.Point(1,1) will raise an unsupported exception.

However, this raises a question: what happens in the case of negation:
-locationHelper.Point(1,1)
Does this return `locationHelper.Point(-1,-1)?
A test to demonstrate this outcome would be good.

feerrenrut · 2018-01-03T01:54:21Z

source/locationHelper.py

+	def __lt__(self,other):
+		"""
+		Returns whether self is less than other.
+		This first compares y, then x coordinates.


I personally find this unintuitive, I would suggest this be a named method rather than using the built in operator. There are several different ways that these comparison operators could work; x then y, y then x, or both must be less. A user of this class may not consider that implementation detail and just rely on the operator, this would lead to subtle and hard to fix bugs. I would prefer to see named methods that convey the differences, this forces the user to consider what it is that they really want.

For instance:

xWiseLessThan(point1, point2)

yWiseLessThan(point1, point2)

xAndYLessThan(point1, point2)

This goes for other comparison methods too.

p = How about Point(1,2)
p.compare(Point.Y_FIRST)
or p.compare(Point.Y_THEN_X)

Hmm, I'm stull a bit puzzled about what names to use. xWiseLessThan sounds ok, but what to use for the le operator? xWiseLessThanOrEqual is a bit long. How about?

yWiseLE

yWiseLT

yWiseGE

yWiseGT

Etc.

@derekriemer: how would less than or equal to fit into your compare idea? Similar to the cmp method, returning 1, 0 or -1? It would be the simplest solution, not very pythonic though IMO.

You could shorten to:

xWiseLessOrEq

xWiseLessThan

xWiseGreaterThan

xWiseGreaterOrEq

I think that is a nice balance between brevity and clarity.
I would prefer not to return 1, 0, -1 from a single compare function.
Another option (borrowing from Derek's idea) would be:
point1.LessThan(Point.xWise, point2)
Though this is longer, with no more clarity. Though probably less lines of code in the Point class required.

I prefer your first suggestion.

feerrenrut · 2018-01-03T01:56:39Z

tests/unit/test_locationHelper.py

+	def test_gt(self):
+		self.assertGreater(Point(x=3,y=4), Point(x=4,y=3))
+
+	def test_lt(self):


For each comparison method please cover other edge cases:

Point is not "comparison type" eg not less than

Point is not "comparison type" but is on the boundary eg not less than (because it is equal)

feerrenrut · 2018-01-03T02:09:57Z

source/locationHelper.py

+				other=toPoint(other)
+			except ValueError:
+				return False
+		return self.x==other.x and self.y==other.y


Is this code limited to working with integers? If working with real numbers is possible / required then some thought will need to be put into this. One method I have seen many times is the Abs(x1 -x2) < EPSILON as a test for equality, a more thorough approach is discussed here: https://stackoverflow.com/a/4029392

I mean this code to work with integers only and I'm quite sure that things would go horribly wrong if you'd use floats and try to convert the resulting rectangle to logical coords or a RECT structure. There is no type checking though.

feerrenrut · 2018-01-03T02:13:31Z

source/locationHelper.py

+		"""Converts self to a L{ctypes.wintypes.POINT}"""
+		return POINT(self.x,self.y)
+
+	def toLogical(self, hwnd):


There is currently nothing to stop you from doing nonsense conversations? For instance p1.toPhysical(hwnd).toPhysical(hwnd)

It could be worth remembering the "type" of the coordinates.

Personally, I'd say this is the responsibility of the person who uses these utilities in his code. Having said that, I've thought about a method to remember the type of coordinates (logical/physical), but I'm afraid that the only way to do this in a reliable manner is adding an extra initialisation argument to the class. I think the extra complexity involved with this is far from elegant.

feerrenrut · 2018-01-03T04:31:06Z

source/locationHelper.py

+			return False
+		return self>other
+
+	def __lt__(self,other):


Again, for the same arguments with point, I don't think operator overriding is a clear way of providing this behaviour.

feerrenrut · 2018-01-03T04:34:05Z

source/locationHelper.py

+
+class Rect(_RectMixin, namedtuple("Rect",("left","top","right","bottom"))):
+	"""Represents a rectangle on the screen.
+	By convention, the right and bottom edges of the rectangle are normally considered exclusive.


Could you clarify what this means with an example? Or better, point to a unit test that demonstrates this.

This is a convention followed by NVDA, as this convension is also followed by Windows. I will document this with references. I'm afraid unit tests do not make sense here.

I would remove the words "normally considered", unless there is some case where they are not exclusive? A unit test could show that given Rect(0, 0, 1, 1) then Point(0,0) is inside, but Point(1,1), Point(0,1), Point(1,0) aren't.

feerrenrut · 2018-01-03T04:37:35Z

source/locationHelper.py

+
+	@property
+	def center(self):
+		return Point((self.left+self.right)/2, (self.top+self.bottom)/2)


Can you add a test for this where self.left = -5, self.right=5, self.top=5, self,bottom=-5

I should have been more specific here. I was worried about 0 divided by 2. You can ignore this.

Thinking about this more carefully, there are other concerns:

A minor one, is performance. Often multiplication is faster than division, though we are unlikely to be doing this so many times an update that this would be at all noticeable.

Rounding, given RectLTRB(0,2,2,0).center() gives Point(1,1) and RectLTRB(0,3,3,0).center() also gives Point(1, 1) callers may be expecting floating point numbers to be returned.

A bug: RectLTRB(10,13,13,10).center() gives Point(1,1) when it should give Point(11,11)
Change the implementation of center to Point( self.left+self.width/2, self.top-self.height/2)
Please add a test for this, and also for the same in the negative quadrant

feerrenrut · 2018-01-03T04:43:20Z

Also, sorry it's taken so long for me to get to reviewing this!

LeonarddeR · 2018-01-03T05:59:48Z

It's not very clear what the difference between Location and Rect is. To me Location sounds like a Point. I think Rect and Location could be named better to highlight their differences.

Any suggestions? I'd say that this naming shouldn't be a huge problem as it is perfectly clear from the classes doc strings what they do exactly. Alternatively, we could rename location to whRect, where w stands for width and h for height, indicating that these rectangles are based on width and height coords. Still, I'd rather have only one of these (preferably only rect), but that would break backwards compatibility.

feerrenrut · 2018-01-03T07:40:09Z

Looking at Rect and Location again, other than creation of the objects is there any reason these could not be merged into one object? The internal representation of an object used to represent a rectangular shape should not matter to the consumer of the class. Width and height apply to both. There would need to be a creation method for creating from Width and Height, something like fromWidthAndHeight

LeonarddeR · 2018-01-03T09:58:00Z

@feerrenrut commented on 3 jan. 2018 08:40 CET:

Looking at Rect and Location again, other than creation of the objects is there any reason these could not be merged into one object?

Yes, backwards compatibility. Object locations are tuples equivalent to Location, the displayModel makes use of rectangle tuples equivalent to Rect.

To give an example, if we'd keep the Rect class and give it a fromLTWH method that creates a new Rect instance from the given lwth, treating the Rect object as a tuple will return (l,t,r,b).

feerrenrut · 2018-01-03T23:41:23Z

Yes, backwards compatibility. Object locations are tuples equivalent to Location, the displayModel makes use of rectangle tuples equivalent to Rect.
To give an example, if we'd keep the Rect class and give it a fromLTWH method that creates a new Rect instance from the given lwth, treating the Rect object as a tuple will return (l,t,r,b).

Ok, I understand now, we do things like top,left,width,height=self.obj.location and I presume also top,left,right,bottom=someRect

In that case, my slight preference would be to make this tuple ordering clear from its typename. It's a bit ugly, but what do you think of:

Rect becoming Rect_TLRB
Location becoming Rect_TLWB

LeonarddeR · 2018-01-04T05:42:03Z

@feerrenrut commented on 4 jan. 2018 00:41 CET:

Ok, I understand now, we do things like top,left,width,height=self.obj.location and I presume also top,left,right,bottom=someRect

Yes.

In that case, my slight preference would be to make this tuple ordering clear from its typename. It's a bit ugly, but what do you think of:

Rect becoming Rect_TLRB

Location becoming Rect_TLWB

I like those, except for the underscores. Do you mind having these removed?

feerrenrut · 2018-01-07T23:57:49Z

Rect becoming Rect_TLRB

Location becoming Rect_TLWB
I like those, except for the underscores. Do you mind having these removed?

Removing the underscores is fine by me. I realise now, I made a mistake the B for the 'Location' replacement name should be a H. This would give:

Rect becomes RectTLRB
Location becomes RectTLWH

…ke much sense

…e__)

…ding according to mathematical rules

feerrenrut · 2018-01-10T04:09:41Z

tests/unit/test_locationHelper.py

+
+class TestToRectLTWH(unittest.TestCase):
+
+	def test_collection(self):


Could you add a comment to explain what this is testing? It isn't obvious to me.

feerrenrut · 2018-01-10T04:10:08Z

tests/unit/test_locationHelper.py

+
+class TestToRectLTRB(unittest.TestCase):
+
+	def test_collection(self):


Could you add a comment to explain what this is testing?

feerrenrut · 2018-01-10T07:35:08Z

source/locationHelper.py

+				other=toRectLTRB(other)
+			except ValueError:
+				return False
+		return other.left == self.left and other.top == self.top and other.right == self.right and other.bottom == self.bottom


Maybe this has already been considered, but it has occured to me that if we are mixing rectangle like classes that come from a source that uses the top left as the origin with those that use the bottom left then given a screen space 10 heigh (indexes 0 to 9):
top of the former is index 3, the top of the latter is index 6 then top should be the same.

I belief we aren't supporting rectangles that are using the bottom left as their origin.

Having said that, wx.Rect seems to be incompatible, so I"m removing that one from the supported classes.

r=wx.Rect(10,10,11,11) >>> r.top 10 >>> r.left 10 >>> r.width 11 >>> r.height 11 >>> r.right 20 >>> r.bottom 20

I don't really understand what is going on in that wx.Rect example? My guess is that right and bottom are inclusive rather than our Rect class which are exclusive?

As I mentioned in my other comment, I think we should add a way to mix coordinate spaces, and force those constructing Rect classes to consider what coordinate space they are in. Even if all the types they are using are in the same space, if you don't consider it you may be visualising things wrong and not get the results that you expect (as I did, even though, with hindsight, I know I should have considered it)

feerrenrut · 2018-01-10T07:44:54Z

source/locationHelper.py

+				other=toRectLTRB(other)
+			except ValueError:
+				return False
+		return other.left >= self.left and other.top >= self.top and other.right <= self.right and other.bottom <= self.bottom


I believe there is a bug in here (both superset and subset ), caused by a mix up of top and bottom comparisons.

Consider:

r1=RectLTRB(left=10, top=14, right=14, bottom=10) r2= RectLTRB(left=11, top=15, right=13, bottom=9)

r1.isSubset(r2) returns False

r1.isSuperset(r2) returns True

r2 in r1 returns True

And:

r3 = RectLTRB(left=10, top=14, right=14, bottom=10) r4 = RectLTRB(left=11, top=13, right=13, bottom=11)

then:

r3.isSubset(r4) returns False

r3.isSuperset(r4) returns False

r4 in r3 returns False

Fix:
For the top and bottom comparisons, other and self should be swapped to the other side of the >= operator. Notice that the __contains__ code for Point has this correct.

For the top and bottom comparisons, other and self should be swapped to the other side of the >= operator. Notice that the contains code for Point has this correct.

The first question I have is, are these rectangles, where the top is lower than the bottom, considered to be valid? If not, I'd go for something like this

Furthermore, could you elaborate on the swapping of self and other? I'm reading this as you mean it to be

return other.left >= self.left and self.top >= other.top and other.right <= self.right and self.bottom <= other.bottom

But that doesn't seem to be valid

How about:

return self.left<=other.left<=other.right<=self.right and self.top<=other.top<=other.bottom<=self.bottom

are these rectangles, where the top is lower than the bottom, considered to be valid?
Whether the top is lower than the bottom is matter of perspective, where 0,0 is the top left and y values decreasing vertically, then yes a top of 14 is below a bottom of 10. However 0,0 in the bottom left and y values increasing vertically, then 14 is above 10.

I hadn't really considered the coordinate space this was supposed to be working in when considering this bug. If I swap the top and bottom values for those rectangles (r1 and r2) the results are as expected.

One way to make the subset and superset functions more robust would be to calculate based on each corner being contained by the rectangle:
return all(p in self for p in [other.topLeft, other.topRight, other.bottomLeft, other.bottomRight])

But I think it might be wise to include the coordinate space with the object. When a rect is created from some other object it can be set automatically, however when it is created by hand, or an unsupported type, then the developer needs to consider the address space.

But I think it might be wise to include the coordinate space with the object.

You mean as a property, and if so, what spaces should we support?

I've been thinking about creating overrides for new in order to allow providing additional parameters, so we can also support setting a property that indicates whether provided coordinates are screen/client and physical/logical.

I think there are really just two likely coordinate systems that will be used:

Top left origin, y increases down the screen.

Bottom left origin, y increases up the screen

I was thinking something like:

# when converting from some other library type, the space does not need to be mentioned r1 = toRectLTWH(someScreenSpaceType) assert(r1.coordSpace == TOP_LEFT_ORIGIN) # when creating manually, the space is mentioned and disambiguates "top" and "bottom", and forces the developer to consider it. r2 = RectLTWH(left=10, top=13, width=5, height=3, coordSpace=BOTTOM_LEFT_ORIGIN) assert(r2.bottom == 10) assert(r2.right == 15) # then same as r2 but other coord system r3 = RectLTWH(left=10, top=13, width=5, height=3, coordSpace=BOTTOM_LEFT_ORIGIN) assert(r3.bottom == 16) assert(r3.right == 15) # Even though the values for bottom do not match, they refer to the same space assert(r2.bottom == r3.bottom) # this is actually a problem. This really depends on the size of the space, whether 10 from the bottom is equal to 16 from the top is really dependent on the size of the screen... maybe things like: r2.subset(r3) # should just result in an exception?

I'm honestly not convinced that we need a system for rectangles based on bottom left origin. Do you have reasons to think that they are currently part of NVDA core or will be in future?

I dont know that we do... but I also dont know that we dont. It's the ambiguity that worries me. I would be fine with saying that this library does not support them, but considering how easy it is to miss a docstring, and make all kinds of mistakes, I would like to see if we can find a way to make it very clear to users of this library how they should be thinking about the values.

How about raising a ValueError when creating a rectangle with right<left and bottom<top for RectLTRB? I'm afraid it is much harder for RectLTWH to enforce clarity, though.

Actually, I have seen an object location of 0, 0, -32000, -32000 in the wild, but these locations don't make sense to me. It looked a bit like giving a location for something we do not know the location of.

A ValueError would be enough for RectLTRB. For the RectLTWH I can't think of anyway to check as it is currently, only with the introduction of an extra construction parameter / or a name change.

…tion test

LeonarddeR · 2018-06-14T06:12:51Z

@feerrenrut: I have fixed another small bug.

I stumbled upon the following. When trying to get bottomRight, bottomLeft or topRight coordinates from a rectangle, some coordinates are exclusive. for bottomRight, both are. For topRight, the x coordinate is exclusive, and for bottomLeft, the y coordiante is exclusive. Should we keep this that way and document it, or should we change this in such a way that these properties return inclusive coordinates? I think the latter might also make sense.

feerrenrut · 2018-06-18T05:12:26Z

For topRight, the x coordinate is exclusive, and for bottomLeft, the y coordinate is exclusive

This does seem to be documented already, right? In the docstring for class RectLTRB. However, I can see how it would be easy to make a mistake when there is a difference in x and y. Having them bundled together as a point gives the indication that what applies to x, should also y, and vice versa. I suggest this is made consistent, probably make all coordinates inclusive.

LeonarddeR · 2018-06-18T06:24:35Z

Hmm, the problem I have with making everything inclusive is that Windows itself as well as the accessibility APIs use exclusive coordinates for the bottom right. That means that we would have to modify every rectangle to make the bottom right point inclusive.

feerrenrut · 2018-06-18T07:23:07Z

Yes, this is a bit annoying. My preference is always with a design that makes it harder to make mistakes. It might require a few changes now, but we are not regularly introducing this kind of code. If we are expecting the same data structure eg RECT to hold both inclusive and exclusive points, then it might be best to provide explicit helpers, or require an explicit argument to the toRectLTRB or toRectLTWH when they are handling this case? Then, we are forcing the developer to think about it when they receive it from the (eg windows) api, and can make guarantees to the consumer of the RectLTRB class

LeonarddeR · 2018-06-18T15:32:52Z

@michaelDCurran, @jcsteh, @dkager, it would be great to have your thoughts about the last 4 comments.

michaelDCurran · 2018-06-18T22:37:20Z

I would have to familiarize myself with this entire PR properly. But the convention on Windows has always been that rects have an exclusive bottom right.
See RECT (Windows): https://msdn.microsoft.com/en-us/library/windows/desktop/dd162897(v=vs.85).aspx
In deed, I don't ever remember coming across a rectangle with inclusive end coordinates in my time on Windows.

feerrenrut · 2018-06-19T00:13:14Z

I probably wasn't clear enough, I am fine with exclusive bottom right coordinate, as long as this is consistent regardless of the source of the rectangle.

@LeonarddeR you mentioned that you "stumbled upon the following." Can you elaborate here? Did this cause a bug, or take some time to debug and understand?

LeonarddeR · 2018-06-19T13:28:28Z

@feerrenrut commented on 19 jun. 2018 02:13 CEST:

@LeonarddeR you mentioned that you "stumbled upon the following." Can you elaborate here? Did this cause a bug, or take some time to debug and understand?

Yeah sure, sorry for being a bit vague there.

For future support of vision enhancement providers, including magnifiers, I've been working on support to get a bounding rectangle for a piece of text using text infos. However, in order for that to work properly, I need to retrieve the first and last offsets of the text that are visible on screen. For that, I was about to use the bottomRight point of the location of the object, until I realised that the bottomRight point is exclusive and therefore belongs to another object.

feerrenrut · 2018-06-25T03:47:21Z

source/NVDAObjects/IAccessible/sysListView32.py

+		# LVM_GETSUBITEMRECT requires a pointer to a RECT structure that will receive the subitem bounding rectangle information.
+		localRect=RECT(
+			left=LVIR_LABEL, # Returns the bounding rectangle of the entire item, including the icon and label
+			top=index # The one-based index of the subitem


Considering this is expecting a one-based index, I think this function should assert that index > 0

feerrenrut · 2018-06-25T03:48:07Z

source/NVDAObjects/IAccessible/sysListView32.py

-		windll.user32.ClientToScreen(self.windowHandle,byref(localRect))
-		windll.user32.ClientToScreen(self.windowHandle,byref(localRect,8))
-		return (localRect.left,localRect.top,localRect.right-localRect.left,localRect.bottom-localRect.top)
+		# ##8268: this might be a malformed rectangle


minor: Too many hash symbols here.

feerrenrut · 2018-06-25T03:52:52Z

source/NVDAObjects/IAccessible/sysListView32.py

-		windll.user32.ClientToScreen(self.windowHandle,byref(localRect,8))
-		return (localRect.left,localRect.top,localRect.right-localRect.left,localRect.bottom-localRect.top)
+		# ##8268: this might be a malformed rectangle
+		# (i.e. with a left coordinate that is greather than the right coordinate).


I'm concerned that this might happen in other cases, can we be more specific about when this is the case? Should location helper generally handle this case? If not, I would like to see that it detects it and alerts us.

Could you add a unit test specifically for malformed rectangles?

Locationhelper detects this and raises a ValueError for RectLTRB. For RectLTWH, there is currently no detection for negative widths.

I'm afraid that we can't make sure that IAccessible or UIA will never return rectangles with negative width and/or height. May be we should no longer raise a ValueError, but log a warning instead.

feerrenrut · 2018-06-25T07:25:02Z

tests/unit/test_locationHelper.py

+			POINT(x=400, y=400),
+			RectLTRB(left=450, top=450, right=490, bottom=990),
+			RECT(450, 450, 490, 990)
+		), rect)


I had to think about this carefully about this, and I'm still not sure I understand this test. How many RectLTRB instances are returned from this call to toRectLTRB? If more than one, which parameters are used? The same questions go for the toRectLTWH version of this test.

toRectLTRB and toRectLTWH always return only one rectangle containing all points in the parameters. For example:

locationHelper.toRectLTRB(locationHelper.Point(3,3), locationHelper.Point(4,4))
RectLTRB(left=3, top=3, right=4, bottom=4)

I realise now that there are still some inconsistencies with regard to inclusivity and exclusivity. toRectLTRB(Point(3,3)) returns RectLTRB(3,3,4,4), and toRectLTRB(Point(3,3), Point(4,4)) returns the same. The reason for this is that toRectLTRB converts a single point to a rectangle with a width and height of 1.

I wondered if it might do something like that. Is this actually used anywhere yet? If not, I suggest we remove it. If it is, then I would prefer a dedicated function with a name that conveys this better eg toRectLTRBContainingAll and has its own documentation.

I realise now that there are still some inconsistencies with regard to inclusivity and exclusivity.

But it sounds like this also needs to be fixed. Perhaps some more tests for this as well.

By the way, there are probably some good names for functions like this in collision detection libraries used in games. You would be looking for something that "creates the minimum volume axis-aligned 2D bounding box".

To make sure, are you suggesting the following way of splitting the functionality of toRectLTRB and TORectLTWH?

Make toRectLTRB and toRectLTWH only do conversion from rect classes and tuples of four integers

Create separate functions to create bounding rectangles from several types of input

…eight 0

LeonarddeR · 2018-07-09T06:10:50Z

@feerrenrut: I've done some major refactoring: I removed the toRectLTRB, toRectLTWH and toPoint helper functions, and split them out into some factory functions on the classes itself. That way, I hope I've made a lot clearer what the several methods ought to do.

…light

feerrenrut

Good work here!

feerrenrut · 2018-07-30T05:50:46Z

source/NVDAObjects/IAccessible/sysListView32.py

@@ -306,6 +306,7 @@ def event_stateChange(self):
 class ListItem(RowWithFakeNavigation, RowWithoutCellObjects, ListItemWithoutColumnSupport):

 	def _getColumnLocationRaw(self,index):
+		assert index>0, "INvalid index: %d" % index


Minor: Too many capitals, the N can be lower case.

feerrenrut · 2018-07-30T06:02:44Z

source/locationHelper.py

 import textInfos
 import wx

 class Point(namedtuple("Point",("x","y"))):
 	"""Represents a point on the screen."""

+	@classmethod
+	def fromNonInts(cls, *nonInts):


Does this work with anything other than a float? I'm concerned that the types that can be used here are are not so obvious. If it is only going to work on floats (and by looks of it also ints) then a name like fromFloatCollection might be better?

Edit:
Upon reading the tests (yay tests), I have realised this is also intended to work with strings. I think this might be a mistake. Firstly, is there anywhere that actually returns coordinates in a string? Second, this would make it all too easy for someone to forget to consider variations in the format of the string due to localisation.

Somehow, It must have been late when I believed that it would make sense to create a rectangle from numbers that are saved in strings, but I guess that doesn't make very much sense. I applied your suggestion.

feerrenrut · 2018-07-30T06:16:56Z

source/locationHelper.py

+		if isinstance(rect,RECT_CLASSES):
+			if cls is RectLTWH:
+				return cls(rect.left, rect.top, rect.right-rect.left, rect.bottom-rect.top)
+			return cls(rect.left, rect.top, rect.right, rect.bottom)


If another type of rect class is introduced this could lead to subtle errors. Instead, do elif cls is RectLTRB: which will allow falling back to raise TypeError

feerrenrut · 2018-07-30T06:17:52Z

source/locationHelper.py

+		if isinstance(point,POINT_CLASSES):
+			if cls is RectLTWH:
+				return cls(point.x, point.y, 0, 0)
+			return cls(point.x, point.y, point.x, point.y)


Also here, be specific to the classes supported.

feerrenrut · 2018-07-30T06:20:25Z

source/locationHelper.py

-				other=toRectLTRB(other)
-			except ValueError:
-				return False
+			return False


Should this be return NotImplemented? or perhaps raise NotImplementedError?

* Created a new locationHelper module

3ad47de

* Added the ClientToScreen function from user32.dll to the winUser module * Use locationHelper for object locations

LeonarddeR requested review from dkager and feerrenrut August 29, 2017 07:08

Leonard de Ruijter added 2 commits September 23, 2017 13:53

Merge remote-tracking branch 'origin/master' into locationHelper

9d84968

Use isinstance instead of type(self) is cls

fb646bd

dkager suggested changes Sep 23, 2017

View reviewed changes

Review actions

8de022d

dkager previously approved these changes Sep 25, 2017

View reviewed changes

feerrenrut suggested changes Jan 3, 2018

View reviewed changes

Leonard de Ruijter added 2 commits January 6, 2018 13:14

Merge remote-tracking branch 'origin/master' into locationHelper

b9d0bac

Class renames

b9c04c5

Leonard de Ruijter added 8 commits January 8, 2018 07:58

Fix __radd__ operator function for Points

1aaa5c0

Added unit test for rect center

e3f8f3a

Rename rectangle operators, remove the sub operator as it does not ma…

7308287

…ke much sense

Rename Point operators

070bb36

Added many more tests, fixed operator overload (__neq__ should be __n…

c2e29fd

…e__)

More unit tests for ctypes, fix __rsub__ for Points

2c9ef24

Fix for getting a center point for a rectangle, including proper roun…

3dbfc75

…ding according to mathematical rules

Last edge case tests

81d3f98

feerrenrut suggested changes Jan 10, 2018

View reviewed changes

Leonard de Ruijter added 3 commits January 10, 2018 12:14

Another implementation for subset and superset, doc string for collec…

0ba335a

…tion test

Remove wx.Rect from supported classes

e49888c

Raise ValueError when left>right and top>bottom for RectLTRB

c6d147f

LeonarddeR dismissed dkager’s stale review via dfc7e0d June 14, 2018 05:08

feerrenrut self-requested a review June 18, 2018 05:12

feerrenrut suggested changes Jun 25, 2018

View reviewed changes

Leonard de Ruijter added 5 commits June 25, 2018 11:24

Review actions for SysListView32

e407922

When a rectangle is created from only one point, make its width and h…

3679aac

…eight 0

Merge remote-tracking branch 'origin/master' into locationHelper

d456b2e

Add fromNonInts factory function

80de132

Use fromNonInts factory function for UIA locations, which are floats

4b4adf3

LeonarddeR mentioned this pull request Jul 5, 2018

Get reliable points from offsets in virtual buffers #8479

Merged

Refactoring

7a05387

Leonard de Ruijter added 4 commits July 18, 2018 08:39

Merge remote-tracking branch 'origin/master' into locationHelper

6c99d8c

Merge remote-tracking branch 'origin/master' into locationHelper

8e808ec

Add expandOrShrink helper method to rectangles, useful for focus high…

5dfca2f

…light

Merge remote-tracking branch 'origin/master' into locationHelper

5d4ec65

feerrenrut reviewed Jul 30, 2018

View reviewed changes

Review actions

f055c9c

feerrenrut approved these changes Jul 30, 2018

View reviewed changes

feerrenrut merged commit 91831b2 into nvaccess:master Jul 30, 2018

nvaccessAuto removed the incubating label Jul 30, 2018

nvaccessAuto added this to the 2018.3 milestone Jul 30, 2018

LeonarddeR added the BabbageWork Pull requests filed on behalf of Babbage B.V. label Oct 11, 2019


		class TestToRectLTWH(unittest.TestCase):

		def test_collection(self):


		class TestToRectLTRB(unittest.TestCase):

		def test_collection(self):

Provide a locationHelper module for code related to locations, rectangles and points #7537

Provide a locationHelper module for code related to locations, rectangles and points #7537

Conversation

LeonarddeR commented Aug 29, 2017

Link to issue number:

Summary of the issue:

Implementation details

Known issues

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feerrenrut left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LeonarddeR Jan 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feerrenrut commented Jan 3, 2018

LeonarddeR commented Jan 3, 2018

feerrenrut commented Jan 3, 2018

LeonarddeR commented Jan 3, 2018

feerrenrut commented Jan 3, 2018

LeonarddeR commented Jan 4, 2018

feerrenrut commented Jan 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LeonarddeR Jan 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LeonarddeR commented Jun 14, 2018

feerrenrut commented Jun 18, 2018

LeonarddeR commented Jun 18, 2018 via email

feerrenrut commented Jun 18, 2018

LeonarddeR commented Jun 18, 2018

michaelDCurran commented Jun 18, 2018

feerrenrut commented Jun 19, 2018

LeonarddeR Jan 8, 2018 •

edited

Loading

LeonarddeR Jan 10, 2018 •

edited

Loading