Fitness distance steps 4 and 5 are inconsistent with the rest of the spec

[guidou]

Steps 4 an 5 refer to settings dictionaries as if they were IDL dictionaries, having members. But settings dictionaries are not IDL dictionaries and their internal structure understood to be an implementation detail elsewhere in the spec.

The spec explicitly says We use the term settings dictionary for the set of values that might be applied as settings to the object., which does not necessarily translate to they working like IDL dictionaries with explicit members that might or might not be present.

Another problem is that step 4 says If constraintValue is a boolean, but the constrainable property is not, which sounds impossible in IDL. I understand that this is intended to support pan/tilt/zoom constraints which accept boolean and numeric values. These constraints are of type (boolean or ConstraintDouble), so the constraint value is compatible with the property. One way to make this clearer is to treat this particular constraint type separately, just like purely numeric constraints are treated separately from purely boolean/enum/string constraints.

[eehakkin]

Steps 4 an 5 refer to settings dictionaries as if they were IDL dictionaries, having members. But settings dictionaries are not IDL dictionaries and their internal structure understood to be an implementation detail elsewhere in the spec.

The spec explicitly says We use the term settings dictionary for the set of values that might be applied as settings to the object., which does not necessarily translate to they working like IDL dictionaries with explicit members that might or might not be present.

Settings dictionaries are not IDL dictionaries but the name settings dictionary still implies that they are dictionaries. A dictionary is commonly defined to be data structure which has keys and values associated to those keys and its main immutable operations are to test the existence of a key and to retrieve the value associated to an existing key.

Another problem is that step 4 says If constraintValue is a boolean, but the constrainable property is not, which sounds impossible in IDL.

It depends on the definition of the constrainable property which I think is not properly defined in the spec.

I understand that this is intended to support pan/tilt/zoom constraints which accept boolean and numeric values. These constraints are of type (boolean or ConstraintDouble), so the constraint value is compatible with the property.

The Terminology section defines Constraints and states that "[f]or each constrainable property, a constraint exists whose name corresponds with the relevant source setting name and capability name". So clearly a constrainable property and a constraint are two different things. Therefore, if a constraint is of type (boolean or ConstraintDouble), it does not mean that the corresponding constrainable property is of that type.

On the other hand, the Constrainable Properties subsection under the MediaStream API section seems to kind of imply that constrainable property and a constraint mean the same.

Could the step 4 be clarified to be If constraintValue is a boolean, but the setting of the property is not? The term the setting of this property is used elsewhere in the spec.

One way to make this clearer is to treat this particular constraint type separately, just like purely numeric constraints are treated separately from purely boolean/enum/string constraints.

Yes, that is one option for the step 4.

But please note that the step 5 is in no way limited to constraints of type (boolean or ConstraintDouble). It affects also for instance the 'focusMode' constraint of type ConstrainDOMString, the 'focusDistance' constraint of type ConstrainDouble and the 'backgroundBlur' constraint of type ConstraintBoolean. The step 5 defines that the fitness distance for an ideal constraint is 1.0 if the underlying device does not support the property or if the settings dictionary does not expose the property. Of course that could be integrated to the steps 7 and 8 were the step 5 to be removed.

[guidou]

Steps 4 an 5 refer to settings dictionaries as if they were IDL dictionaries, having members. But settings dictionaries are not IDL dictionaries and their internal structure understood to be an implementation detail elsewhere in the spec.
The spec explicitly says We use the term settings dictionary for the set of values that might be applied as settings to the object., which does not necessarily translate to they working like IDL dictionaries with explicit members that might or might not be present.

Settings dictionaries are not IDL dictionaries but the name settings dictionary still implies that they are dictionaries. A dictionary is commonly defined to be data structure which has keys and values associated to those keys and its main immutable operations are to test the existence of a key and to retrieve the value associated to an existing key.

If they're not necessarily IDL dictionaries we shouldn't link to the definition of IDL dictionaries when talking about existence of fields. The implementation of SelectSettings does not require that we have a list of IDL Settings dictionaries that are checked individually (and, for example, the Chromium implementation doesn't follow that approach). It can be the case that support for a particular property is determined by calling some system API and taking a code path that covers multiple dictionaries simultaneously.

Another problem is that step 4 says If constraintValue is a boolean, but the constrainable property is not, which sounds impossible in IDL.

It depends on the definition of the constrainable property which I think is not properly defined in the spec.

I would say that it is. The term constrainable property refers to a name (and type) associated with a constrainable object, which has Capabilities, Constraints and Settings. There is a an IDL template for these concepts. See https://w3c.github.io/mediacapture-main/#interface-definition-0

I understand that this is intended to support pan/tilt/zoom constraints which accept boolean and numeric values. These constraints are of type (boolean or ConstraintDouble), so the constraint value is compatible with the property.

The Terminology section defines Constraints and states that "[f]or each constrainable property, a constraint exists whose name corresponds with the relevant source setting name and capability name". So clearly a constrainable property and a constraint are two different things. Therefore, if a constraint is of type (boolean or ConstraintDouble), it does not mean that the corresponding constrainable property is of that type.

On the other hand, the Constrainable Properties subsection under the MediaStream API section seems to kind of imply that constrainable property and a constraint mean the same.

Since constrainable objects are a pattern they don't have a single explicit type, but it is easy to see that they have an implicit type that is expressed in different ways for settings, constraints and capabilities.
For example, in the case of frameRate, its settings are of type double, its capabilties is of type DoubleRange, and constraints are of type ConstraintDouble. It is impossible to have a string to express a constraint, setting or capability for frameRate.

Since properties are a pattern and not a specific IDL definition (beyond the name of the property), sometimes the spec uses the setting type when describing the property and sometimes it uses the constraint type. I agree that the spec should be more consistent here, but that doesn't change the fact that the types for settings, capabilities and constraints have to be compatible and you can't have an unrelated type as one of its values.

Could the step 4 be clarified to be If constraintValue is a boolean, but the setting of the property is not? The term the setting of this property is used elsewhere in the spec.

This doesn't make sense because if you define a type in IDL (and you have to do it for constraints) the only possible values have to correspond to that type. Having a step to cover the case where the value does not correspond to its type is unnecessary/incorrect since that value wouldn't exist.

One way to make this clearer is to treat this particular constraint type separately, just like purely numeric constraints are treated separately from purely boolean/enum/string constraints.

Yes, that is one option for the step 4.

Let's go for it then.

But please note that the step 5 is in no way limited to constraints of type (boolean or ConstraintDouble). It affects also for instance the 'focusMode' constraint of type ConstrainDOMString, the 'focusDistance' constraint of type ConstrainDouble and the 'backgroundBlur' constraint of type ConstraintBoolean. The step 5 defines that the fitness distance for an ideal constraint is 1.0 if the underlying device does not support the property or if the settings dictionary does not expose the property. Of course that could be integrated to the steps 7 and 8 were the step 5 to be removed.

There are already definitions of fitness distance for strings, doubles and booleans, so I see no need to define anything different for those properties. If these properties should be treated differently to other booleans, strings and doubles they should have a different type and we can make fitness distance definitions for those types.

This concept of having the settings dictionary member exist or not just introduces unnecessary complexity based on things that are either impossible or impractical and we should remove it. Why do we need anything beyond the concept of the setting dictionary not supporting the constraint value ?

IMO, we should just return to the old fitness distance definition and just add an extra step on how to calculate its value for the (boolean or ConstraintDouble) type.

Why do you think we should do anything different from that?

[eehakkin]

If they're not necessarily IDL dictionaries we shouldn't link to the definition of IDL dictionaries when talking about existence of fields.

True.

The implementation of SelectSettings does not require that we have a list of IDL Settings dictionaries that are checked individually (and, for example, the Chromium implementation doesn't follow that approach). It can be the case that support for a particular property is determined by calling some system API and taking a code path that covers multiple dictionaries simultaneously.

That is usually also the only practical implementation because the total number of all possible settings dictionaries is the product of number of all possible settings for each property supported by the implementation and the source.

Could the step 4 be clarified to be If constraintValue is a boolean, but the setting of the property is not? The term the setting of this property is used elsewhere in the spec.

This doesn't make sense because if you define a type in IDL (and you have to do it for constraints) the only possible values have to correspond to that type. Having a step to cover the case where the value does not correspond to its type is unnecessary/incorrect since that value wouldn't exist.

MediaTrackConstraintSet WebIDL defines pan constraint to be of type (boolean or ConstrainDouble) so constraintValue for pan can be a boolean where as MediaTrackSettings WebIDL defines pan setting to be of type double so it definitely is not a boolean. So the value does correspond its type but not its setting's type. IMHO that makes sense.

There are already definitions of fitness distance for strings, doubles and booleans, so I see no need to define anything different for those properties.

For all positive numeric constraints, the fitness distance is the result of the formula
(actual == ideal) ? 0 : |actual - ideal| / max(|actual|, |ideal|).
What if a site calls navigator.mediaDevices.getUserMedia({iso: 400}) and there is a source which does not support ISO? An implementation should compute (or simulate to compute) the fitness distance between the constraints and all possible settings dictionaries. It is clear that ideal is 400 in this case but what is actual? There is no actual ISO. The source does not support ISO. So that definition would need an addition stating that the fitness distance in this case is 1 because there are no actual value. But that is exactly what the step 5 tries to do.

Of course, it is possible to remove the step 5 and integrate it in and repeat at steps 7 and 8.

IMO, we should just return to the old fitness distance definition and just add an extra step on how to calculate its value for the (boolean or ConstraintDouble) type.

Wouldn't that basically be a combination of the current steps 4 and 7?

[guidou]

Could the step 4 be clarified to be If constraintValue is a boolean, but the setting of the property is not? The term the setting of this property is used elsewhere in the spec.

This doesn't make sense because if you define a type in IDL (and you have to do it for constraints) the only possible values have to correspond to that type. Having a step to cover the case where the value does not correspond to its type is unnecessary/incorrect since that value wouldn't exist.

MediaTrackConstraintSet WebIDL defines pan constraint to be of type (boolean or ConstrainDouble) so constraintValue for pan can be a boolean where as MediaTrackSettings WebIDL defines pan setting to be of type double so it definitely is not a boolean. So the value does correspond its type but not its setting's type. IMHO that makes sense.

But the concept we are interested in is whether a dictionary setting satisfies a constraint or not, see Step 2. Thus, having a numeric setting and a boolean value for the constraint doesn't mean that the setting cannot be checked to satisfy the constraint. IIUC, if the setting supports any numeric value it means it satisfies a true value for the constraint.
I see no need to add additional language there other than specifying how to compute the numeric values for the fitness in a step specific to (boolean or ConstraintDouble).

There are already definitions of fitness distance for strings, doubles and booleans, so I see no need to define anything different for those properties.

For all positive numeric constraints, the fitness distance is the result of the formula (actual == ideal) ? 0 : |actual - ideal| / max(|actual|, |ideal|). What if a site calls navigator.mediaDevices.getUserMedia({iso: 400}) and there is a source which does not support ISO? An implementation should compute (or simulate to compute) the fitness distance between the constraints and all possible settings dictionaries. It is clear that ideal is 400 in this case but what is actual? There is no actual ISO. The source does not support ISO. So that definition would need an addition stating that the fitness distance in this case is 1 because there are no actual value. But that is exactly what the step 5 tries to do.

I see your point. The phrasing looked very confusing to me. I think we should phrase it differently for extra clarity. Something like If an ideal value is specified and the settings dictionary cannot support any value for the property, the fitness distance is 1 .

Of course, it is possible to remove the step 5 and integrate it in and repeat at steps 7 and 8.

Now that I see the intent, I think would keep it, but with the rephrasing above (or similar).

IMO, we should just return to the old fitness distance definition and just add an extra step on how to calculate its value for the (boolean or ConstraintDouble) type.

Wouldn't that basically be a combination of the current steps 4 and 7?

My summary is that we should:

• Eliminate Step 4
• Keep the current Step 5, but rephrase it for extra clarity.
• Add a step to explain how to compute the fitness distance for (boolean or ConstraintDouble) constraints.

Do you agree?

[eehakkin]

But the concept we are interested in is whether a dictionary setting satisfies a constraint or not, see Step 2. Thus, having a numeric setting and a boolean value for the constraint doesn't mean that the setting cannot be checked to satisfy the constraint. IIUC, if the setting supports any numeric value it means it satisfies a true value for the constraint. I see no need to add additional language there other than specifying how to compute the numeric values for the fitness in a step specific to (boolean or ConstraintDouble).

Oh, I see now that the step 4 is worded so that it applies both to required boolean (boolean or ConstraintDouble) constraints (a boolean constraint is required in advanced constraint sets as you know) and to ideal boolean (boolean or ConstraintDouble) constraints (a boolean constraint is ideal in the basic constraint set as you know).

That should be fixed, I think.
I agree that there is no need to add additional language for the required case.

I see your point. The phrasing looked very confusing to me. I think we should phrase it differently for extra clarity. Something like If an ideal value is specified and the settings dictionary cannot support any value for the property, the fitness distance is 1 .

Of course, it is possible to remove the step 5 and integrate it in and repeat at steps 7 and 8.

Now that I see the intent, I think would keep it, but with the rephrasing above (or similar).

Good.

My summary is that we should:

• Eliminate Step 4
• Keep the current Step 5, but rephrase it for extra clarity.
• Add a step to explain how to compute the fitness distance for (boolean or ConstraintDouble) constraints.

Do you agree?

Mostly, but I still think that the current step 4 has value for the ideal case.

While at it, I noticed that the current step 6 has probably too low priority.

So would something like below do?

---

1. If constraintName is not supported by the User Agent, the fitness distance is 0.

2. If the constraint is required (constraintValue either contains one or more members named 'min', 'max', or 'exact', or is itself a bare value and bare values are to be treated as 'exact'), and the settings dictionary's constraintName member's value does not satisfy the constraint or doesn't exist, the fitness distance is positive infinity.

3. If the constraint does not apply for this type of object, the fitness distance is 0 (that is, the constraint does not influence the fitness distance).

4. If no ideal value is specified (constraintValue either contains no member named 'ideal', or, if bare values are to be treated as 'ideal', isn't a bare value), the fitness distance is 0.

5. If constraintValue is a boolean, but the constrainable property is not (the (boolean or ConstraintDouble) case), then the fitness distance is based on whether settings dictionary supports any value for the property or not, from the formula

   (actualSupportForAnyValue == ideal) ? 0 : 1

6. If the settings dictionary cannot support any value for the property, the fitness distance is 1.

7. For all numeric constraints (such as height, width, frameRate, aspectRatio, sampleRate and sampleSize), the fitness distance is the result of the formula

   (actual == ideal) ? 0 : |actual - ideal| / max(|actual|, |ideal|, |actual - ideal|)

8. For all string, enum and boolean constraints (e.g. deviceId, groupId, facingMode, resizeMode, echoCancellation), the fitness distance is the result of the formula

   (actual == ideal) ? 0 : 1

---

I used your wording for my step 6 but without If an ideal value is specified because that is taken care of by my step 4.

It would be possible to replase my step 5 with a comprehensive (boolean or ConstrainDouble) case but that would have to above my step 6 and it would have to repeat my steps 6 and 7, which is suboptimal IMHO.

[eehakkin]

One option is also to change the meaning of a pan/tilt/zoom: false constraint.

My reading of the current fitness distance algorithm is that pan: true is an exact (in advanced constraint sets) or ideal (in the basic constraint set) constraint for a pan-capable device whereas pan: false is an exact (in advanced constraint sets) or ideal (in the basic constraint set) constraint for a pan-incapable device. That kind of makes sense. False and true are antonyms and so are pan-capable and pan-incapable.

But usually it is not question about whether a site wants a pan-capable device or a pan-incapable device but whether a site wants a pan-capable device or not (i.e. it does not matter whether the device is pan-capable or pan-incapable). So it might make more sense to refine the fitness distance algorithm so that pan: true would still be an exact (in advanced constraint sets) or ideal (in the basic constraint set) constraint for a pan-capable device but pan: false would not constrain anything (similar to that facingMode: [] does not constrain anything). That would also match the behavior of the the only current implementation (Chrome).

This would also simplify the fitness distance algorithm as the (boolean or ConstrainDouble) case could be handled fully after the If the settings dictionary cannot support any value for the property, the fitness distance is 1. step.

[alvestrand]

Not specifying the fitness distance constraint is a standard and effective means of saying "no preference".

[jan-ivar]

Steps 4 an 5 refer to settings dictionaries as if they were IDL dictionaries, having members. But settings dictionaries are not IDL dictionaries and their internal structure understood to be an implementation detail elsewhere in the spec.
...
If they're not necessarily IDL dictionaries we shouldn't link to the definition of IDL dictionaries when talking about existence of fields.

There's no reference to WebIDL in step 4 or 5:

4. "If constraintValue is a boolean, but the constrainable property is not, then the fitness distance is based on whether the settings dictionary's constraintName member exists or not".
5. "If the settings dictionary's constraintName member does not exist, the fitness distance is 1."

These link to https://infra.spec.whatwg.org/#map-exists, making settings dictionary an ordered map, a common data structure recommended to describe a set of named values (key-value pairs) internally. It's not WebIDL.

WebIDL dictionaries also happen to be "used to define an ordered map data type", but that doesn't make this WebIDL.

We could perhaps clarify this better in the definition of settings dictionary:

- We use the term settings dictionary for the set of values that might be applied as
+ We use the term settings dictionary for the ordered map of values that might be applied as
  settings to the object.

Another problem is that step 4 says If constraintValue is a boolean, but the constrainable property is not, which sounds impossible in IDL. I understand that this is intended to support pan/tilt/zoom constraints which accept boolean and numeric values. These constraints are of type (boolean or ConstraintDouble), so the constraint value is compatible with the property.

Only constraintValue here is from WebIDL (coming from CS), and it can be either a boolean or a ConstraintDouble.

But the constrainable property is not WebIDL, and refers instead to the property being constrained by the constrainable pattern. E.g. Its type in the case of pan/tilt/zoom is double.

[jan-ivar]

Since properties are a pattern and not a specific IDL definition (beyond the name of the property), sometimes the spec uses the setting type when describing the property and sometimes it uses the constraint type. I agree that the spec should be more consistent here

Agreed, and thanks for spotting this! I've filed #945 to address this.

Regarding the remainder of the conversation (which has gotten long), to scope it down, it seems centered on confusion regarding #766, is that fair?

But the concept we are interested in is whether a dictionary setting satisfies a constraint or not, see Step 2. Thus, having a numeric setting and a boolean value for the constraint doesn't mean that the setting cannot be checked to satisfy the constraint. IIUC, if the setting supports any numeric value it means it satisfies a true value for the constraint.

Here's step 2 today: "If the constraint is required (constraintValue either contains one or more members named 'min', 'max', or 'exact', or is itself a bare value and bare values are to be treated as 'exact'), and the settings dictionary's constraintName member's value does not satisfy the constraint or doesn't exist, the fitness distance is positive infinity."

What "satisfies" (before #766) was considered self-evident by language like "The maximum legal value of this property." (opening another issue #946, sigh).

#766 added the "or doesn't exist" part, which perhaps requires clarification.

Our thinking here was that during getUserMedia especially, the fitness algorithm needs to take into account devices that have pan/tilt/zoom and those that don't. The former devices would have pan/tilt/zoom members in their settings dictionaries, while the latter devices would not.

The "or doesn't exist" part should be clear that pan: true results in infinity for the latter devices.

But I agree it is perhaps unclear that the settings dictionaries of the former devices "satisfy" pan: true.

I'm happy to take language to clarify that sentence. I'd like to keep changes here limited however, as I don't think a major refactor of the algorithm is needed here, or desirable if we can avoid it.

[jan-ivar]

One option is also to change the meaning of a pan/tilt/zoom: false constraint.

Please open a separate issue if we wish to rehash pan: false from w3c/mediacapture-image#256.

[eehakkin]

One option is also to change the meaning of a pan/tilt/zoom: false constraint.

Please open a separate issue if we wish to rehash pan: false from w3c/mediacapture-image#256.

For me the current behavior is fine and Harald seemed to be against the change (#933 (comment)) so let's not change it and therefore let's not open an issue.

[guidou]

I think the issue can be resolved if we are more explicit and consistent at defining a few things.

Some examples of ambiguity:

• In Step 3 of applyConstraints it says that a settings dictionary "refers to a possible instance of the MediaTrackSettings dictionary", which is IDL. But settings dictionary are also defined as "the set of values that might be applied as settings to the object", which is a more flexible definition where the only important concept is supporting a constraint or not, regardless of the implementation of the dictionary. If we do decide that a settings dictionary is an instance of the MediaTrackSettings WebIDL type, that's fine, but then let's be explicit about it everywhere, and about what not having a field means.

• The expression "type of a constrainable property". A constrainable property is a pattern that has settings, constraints and capabilities. The types for those three things can be different, but whenever we have a constraint value coming from WebIDL, that value is valid (as a constraint) even if it is not the same type of the setting. Saying that it is not of the same type of the property sounds misleading as it suggests that it is possible to specify a constraint value that is invalid or incompatible for the property. Nowhere in the spec it is established that the type of a property is the type of its setting. Moreover, it is not immediately obvious to me that the type of a property whose constraints are (boolean or ConstraintDouble) is the same as the type of a property whose constraints are ConstraintDouble, even if settings are of type double for both of them. Maybe we can define somewhere that the expression "type of a constraintable property" is a shorthand for the type of its setting.

• I think the case of the fitness distance for (boolean or ConstraintDouble) constraints would be more readable with an explicit step for that constraint type, similar to how steps 7 and 8 are handled for other constraint types rather than having an early step to handle the case where the constraint value is boolean but the property is not, especially if we are not explicit about what the type of a property is.

All this said, I think the interaction between constraint values and dictionary settings only require the concept of a dictionary setting satisfying a constraint value or not, and letting implementations decide how dictionary settings are implemented and how they satisfy a constraint value. But I agree that defining dictionary settings as instances of MediaTrackSettings is probably the way that requires fewer changes.