How to calculate FI-DockQ and API-DockQ?

[bbjy]

Hi @amorehead, thank you for your work! Would you please explain that how to calculate FI-DockQ and API-DockQ? Or it will be very helpful to release the code of metrics for evaluating the refined models.

Moreover, what are the distance thresholds of inter-chain contact (or interface) for calculating Fnat and iRMSD?

Thank you so much!

[amorehead]

Hi, @bbjy. I apologize for my delay in responding to your question. FI-DockQ represents the fraction of protein structures for which a method was able to improve upon the DockQ score (i.e., nativeness) of the original input structure. For example, in a test dataset of 5,000 protein structures, a method might be able to improve the DockQ scores of 30% of them, regardless of the size of the improvement. Similarly, API-DockQ measures the average improvement to DockQ scores a method was able to make using a set of input protein structures.

We are planning to release the remainder of the source code for DeepRefine soon, so our method for calculating these metrics should soon be available. However, in short, we opted to record each method's change in DockQ scores for a test dataset's input proteins within individual CSV files and then run a simple Python analysis script on a collection of these CSV files to aggregate average results between different training runs for FI-DockQ and API-DockQ.

When it comes to the distance thresholds, I do not recall changing them from the default values the DockQ program sets for users. I will refer you to DockQ to see which distance threshold they ultimately use for Fnat and iRMSD. If I'm not mistaken, I believe they use a threshold of 8 Angstrom.

[bbjy]

Thank you for your reply! @amorehead
I think I know how to calculate the two metrics. I try to test the released models in folder "project/checkpoints/EGR_All_Atom_Models" on Benchmark_2 test dataset as you released. And then calculating the FI-DockQ and API-DockQ with the following code. But I don't get the similar result with your paper. Is there anything wrong in the following code?
Thanks very much! Looking forward to your complete source code for DeepRefine!

`

FI-DockQ

total = len(decoy_dockQ_list)
num1 = len(np.where((np.array(refined_dockQ_list1) - np.array(decoy_dockQ_list))>0)[0])
fi_dockq1 = num1 * 1.0 / total
num2 = len(np.where((np.array(refined_dockQ_list2) - np.array(decoy_dockQ_list))>0)[0])
fi_dockq2 = num2 * 1.0 / total
num3 = len(np.where((np.array(refined_dockQ_list3) - np.array(decoy_dockQ_list))>0)[0])
fi_dockq3 = num3 * 1.0 / total
avg_fi_dockq = np.mean([num1, num2, num3])
var_fi_dockq = np.var([num1, num2, num3])
print('avg_fi_dockq: ', avg_fi_dockq, ' var_fi_dockq: ', var_fi_dockq)

API-DockQ

api1 = np.mean(np.array(refined_dockQ_list1) - np.array(decoy_dockQ_list))
api2 = np.mean(np.array(refined_dockQ_list2) - np.array(decoy_dockQ_list))
api3 = np.mean(np.array(refined_dockQ_list3) - np.array(decoy_dockQ_list))
avg_api_dockq = np.mean([api1, api2, api3])
var_api_dockq = np.var([api1, api2, api3])
print('avg_api_dockq: ', avg_api_dockq, " var_api_dockq: ", var_api_dockq)`