Mismatch in Arguments for eps_net and Issues with Test Results Reproduction

[BL-Lac149597870]

Hello,

I encountered an issue with the function eps_net during the inference process here L274-L277. It appears that the number of arguments expected by the forward() method of eps_net does not match the number passed in the code.

Upon inspecting the code, I found that the argument R_t_global was not being passed. I tried to fix this by adding the following line, mimicking the training part of the code:

_, R_t_global = global_frame(X_t, mask_gen_pos)

and then passed the variable R_t_global to the eps_net as:

vp_t, vr_t, vd_t, vc_t = self.eps_net(
                    d_t, s_t, X_t, R_t, R_t_global, res_feat, pair_feat, t_tensor, 
                    mask_gen_d, mask_gen_aa, mask_gen_pos, mask_res
                )   # (N, L, 3), (N, L, 3, 3), (N, L, 3)

As a result, the sample function looks like this:

X_t = manifold_to_euclid(r_t, p_t, d_t, X_ctx, mask_gen_pos)

# WARN: fix missed R_t_global
_, R_t_global = global_frame(X_t, mask_gen_pos)
# ===========================

X_t, R_t = X_t[:, :, BBHeavyAtom.CA], construct_3d_basis(X_t[:, :, BBHeavyAtom.CA],
                                                         X_t[:, :, BBHeavyAtom.C],
                                                         X_t[:, :, BBHeavyAtom.N],)

vp_t, vr_t, vd_t, vc_t = self.eps_net(
        d_t, s_t, X_t, R_t, R_t_global, res_feat, pair_feat, t_tensor, 
        mask_gen_d, mask_gen_aa, mask_gen_pos, mask_res
    )   # (N, L, 3), (N, L, 3, 3), (N, L, 3)         

1、Could you please confirm if the changes I made to the code are correct?

---

I then used this fix to generate peptides with the test dataset using pretrained weights from the ppflow_pretrained.pt ckpt and evaluated the generated peptides(bb4.pdb format) with evaluation/eval_struct_seq.py. However, the results I obtained did not correspond with the results presented in your paper.

Model	ΔG(↓)	IMP%-B(↑)	IMP%-S(↑)	Validity(↑)	Novelty(↑)	Diversity
PPFLOW-BB	-349.59	36.02%	10.34%	1.00	0.84	0.76
reproduction	-	-	0.04%	1.00	0.97	0.64

If it helps, corresponding raw files and evaluation meta files can be downloaded here.

2、Could there be any additional minor errors in the code that are preventing the results from being accurately reproduced?

I really appreciate your help in resolving this issue. Thank you for your continued support and dedication to improving this project!

[EDAPINENUT]

Thank you for raising the issue. We are working on resolving it. Due to a recent code reorganization, the model was retrained, and we have noticed many discrepancies compared to the initial model. Additionally, there have been occurrences of NaN during training. We will conduct a thorough review and verify the samples from the initial dataset.

[EDAPINENUT]

Besides, thanks a lot for your interest in our work. Please feel free to send an e-mail to me for a detailed discussion ([email protected]), or directly add my WeChat.

[EDAPINENUT]

I have uploaded my previously generated peptides, and evaluated them again. (100/structure)
The evaluated results are

IMP-S: 12.50%
Validity: 1.00
Nov: 0.99
div: 0.92

You can download codesign_results.tar.gz from the google drive.

Besides, we use _bb3.pdb for evaluation because, in comparison with diffpp, it just generates 3 backbone atoms. You can filter the file name ending with _bb3.pdb and try the evaluation again.

We will also check the previous checkpoints and upload it again.

[BL-Lac149597870]

Thank you very much for providing the original peptide files! Upon careful review, I've encountered several issues:

• Validity Evaluation Code Bug: The code used to evaluate validity contains a bug; it should utilize logical_or instead of logical_and here.

• Peptide Validity drops to zero after Correction: Following the correction of the validity calculation, it appears that almost all the provided peptides are invalid due to the amino acid atoms being too far apart. This condition leads to anomalies(fake higher IMP_S) during the FoldX energy calculations. It seems that FoldX is unable to provide reasonable scores for fragmented peptides. I suspect that the peptide coordinates might have been altered, given that peptides designed by your program typically should not exhibit invalid structures.

• Reevaluation of FoldX Energy: I reevaluated the FoldX energy using the suggested bb3.pdb files, which are peptides newly sampled using your trained model and sampling codes (ppflow-bb mode). I then recalculated the IMP_S metric, unfortunately finding that it still remains at a low 0.04%.

Thank you again for your assistance with this research! I really appreciate your assistance in resolving this issue.

[EDAPINENUT]

We have identified this strange phenomenon and are committed to resolving it. Since our reconstruct algorithm is based on angles, the bond lengths should theoretically remain mostly fixed. However, there are indeed some issues in the uploaded samples. We will resolve this shortly and provide a response soon.

[EDAPINENUT]

We have identified this strange phenomenon and are committed to resolving it. Since our reconstruct algorithm is based on angles, the bond lengths should theoretically remain mostly fixed. However, there are indeed some issues in the uploaded samples. We will resolve this shortly and provide a response soon.