Incorporating inequality constraints

[lonssi]

I am currently working on a project where I am using symbolic regression to define a function based on a set of input-output pairs. The function produces a single output value.

In some cases, I know the exact output value that the function should produce for a given set of input parameters, but in other cases, I only know the relative ordering of the output values for different sets of input parameters. For example, I know that the function should produce a higher output value for one set of input parameters (x1, y1, z1, ...) than for another set of input parameters (x2, y2, z2, ...).

I am facing some difficulties in incorporating this type of inequality constraints into the optimization process. Any ideas or suggestions on how to approach this problem?

Thank you in advance for your help!

[MilesCranmer]

You should be able to write a custom loss function that does this for you. (Losses need not be differentiable!). See https://astroautomata.com/PySR/options/#loss for some examples. Note that you can define very complex functions as losses as any Julia code is allowed.

[MilesCranmer]

For example, if you define the “weights” you pass as simply: if weight = 1, you want higher, if weight = 0, you want lower. Then, you could do:

loss="""myloss(x, y, w) = begin
    if w == 1
        if x > y
            return 1
        else
            return 0
        end
    else
        if x > y
            return 0
        else
            return 1
        end
    end
end"""

Pass this a loss parameter to PySRRegressor, and then when calling .fit(), pass the weight vector indicating what behavior you want at each point.

[lonssi]

Thanks for the example, but I can't quite wrap my head around it.

I'll provide here a toy example to make sure that there's no misunderstanding:

input1 = [0.2, 0.6]
output1 = 0.5
weight1 = 1

input2 = [0.1, 0.3]
output2 = 0.1
weight2 = 1

input3 = [0.7, 0.1]
output3 = 0.9
weight3 = 1

X = np.array([input1, input2, input3])
y = np.array([output1, output2, output3])
w = np.array([weight1, weight2, weight3])

model = PySRRegressor(loss=loss_str)

model.fit(X, y, weights=w)

For all of the inputs 1-3, I am certain what the corresponding output should be.

However, for the following inputs, I don't know what the output should exactly be:

input4 = [0.0, 0.1]
input5 = [0.2, 0.4]
input6 = [0.0, 1.0]

But I do know that the following assertion must hold:
model.predict(input4) >= model.predict(input5) >= model.predict(input6)

When fitting the model, I want to use inputs 1-3 and also the inequality constraint. For the outputs that I'm certain of, I would like to use square deviation loss. Can the example you provided be used in this context? I can't see how.

[MilesCranmer]

I see. This constraint: model.predict(input4) >= model.predict(input5) >= model.predict(input6), or any loss function that is not computable by summing over all samples, takes more work. However, it's definitely doable.

Have a look at https://astroautomata.com/PySR/backend/ for advice on how to set this up.