split of code between FCCAnalyses and FCCeePhysicsPerformance

[jalimena]

@EmanuelPerez @patriziaazzi

We have two repositories, FCCAnalyses and FCCeePhysicsPerformance, and FCCeePhysicsPerformance is very dependent on FCCAnalyses. Seemingly every time that FCCAnalyses is updated, FCCeePhysicsPerformance also needs a corresponding change. This has been an issue that has come up several times now, and it really grinds analysis to a halt as we try to figure out what exactly needs to be synced between the two repositories. Is there a way we could somehow merge the two repositories, or at least make them less dependent on each other?

@clementhelsens and I have discussed this, and I think we agree this part of the workflow could be significantly improved

See also https://fccsw-forum.web.cern.ch/t/solved-port-recent-fccanalyses-compilation-fixes-into-fcceephysicsperformance/107

[patriziaazzi]

Dear Juliette, Indeed what you are asking makes a lot of sense and I thought it would have been the plan all along. In my mind, the FCCPhysicsPerformance directories are a place where people keep the documentation relative to the Case Studies, including some code that is specific and maybe even not so dependent on FCCAnalysis. More like a “working group” space that could have instant documentation via git, and not the main place for the code. Maybe some sort of better “Project management” software could be used for the future once the number of case studies starts increasing considerably… Conversely I had imagined that in FCCAnalysis we could start having a structure as well representing the case studies that would include more “mature” code. So in some sense, indeed, I would merge the more “coding” content of the Performance directory into the FCCAnalysis (to facilitate all these compilation issues in fact), while leaving maybe a similar naming structure so that it could be easier for people to navigate to the code related to a specific case study. I agree this should be discussed urgently. Best, Pat On 28 Apr 2022, at 15:37, Juliette Alimena ***@***.******@***.***>> wrote: @EmanuelPerez<https://github.com/EmanuelPerez> @patriziaazzi<https://github.com/patriziaazzi> We have two repositories, FCCAnalyses and FCCeePhysicsPerformance, and FCCeePhysicsPerformance is very dependent on FCCAnalyses. Seemingly every time that FCCAnalyses is updated, FCCeePhysicsPerformance also needs a corresponding change. This has been an issue that has come up several times now, and it really grinds analysis to a halt as we try to figure out what exactly needs to be synced between the two repositories. Is there a way we could somehow merge the two repositories, or at least make them less dependent on each other? @clementhelsens<https://github.com/clementhelsens> and I have discussed this, and I think we agree this part of the workflow could be significantly improved See also https://fccsw-forum.web.cern.ch/t/solved-port-recent-fccanalyses-compilation-fixes-into-fcceephysicsperformance/107 — Reply to this email directly, view it on GitHub<#40>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABKTDTZJXQ5EQSJKCZUATNDVHKICHANCNFSM5USOA2EA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[EmanuelPerez]

There were reasons for setting up things like that - with Clement - more than one year ago. The PhysPerf repository was supposed to contain the analysis-dependent code, things that are very specific to a given analysis and need not be in a central place. This code may not be dependent on FCCAnalysis, but chances are that most of it will be. Also, we thought of it as the place where, besides the documentation, "all what is needed to reproduce a given analysis" would be kept. One could instead have everything within the FCCAnalysis repo - with some subdirectories that separate what is the "core" from what is analysis specific. But before making changes in a rush, one should think whether this will scale since, basically, it means that you will compile your analysis code together with the analysis code of everyone else. E. On 28 Apr 2022, at 16:08, Patrizia Azzi ***@***.******@***.***>> wrote: Dear Juliette, Indeed what you are asking makes a lot of sense and I thought it would have been the plan all along. In my mind, the FCCPhysicsPerformance directories are a place where people keep the documentation relative to the Case Studies, including some code that is specific and maybe even not so dependent on FCCAnalysis. More like a “working group” space that could have instant documentation via git, and not the main place for the code. Maybe some sort of better “Project management” software could be used for the future once the number of case studies starts increasing considerably… Conversely I had imagined that in FCCAnalysis we could start having a structure as well representing the case studies that would include more “mature” code. So in some sense, indeed, I would merge the more “coding” content of the Performance directory into the FCCAnalysis (to facilitate all these compilation issues in fact), while leaving maybe a similar naming structure so that it could be easier for people to navigate to the code related to a specific case study. I agree this should be discussed urgently. Best, Pat On 28 Apr 2022, at 15:37, Juliette Alimena ***@***.******@***.***>> wrote: @EmanuelPerez<https://github.com/EmanuelPerez> @patriziaazzi<https://github.com/patriziaazzi> We have two repositories, FCCAnalyses and FCCeePhysicsPerformance, and FCCeePhysicsPerformance is very dependent on FCCAnalyses. Seemingly every time that FCCAnalyses is updated, FCCeePhysicsPerformance also needs a corresponding change. This has been an issue that has come up several times now, and it really grinds analysis to a halt as we try to figure out what exactly needs to be synced between the two repositories. Is there a way we could somehow merge the two repositories, or at least make them less dependent on each other? @clementhelsens<https://github.com/clementhelsens> and I have discussed this, and I think we agree this part of the workflow could be significantly improved See also https://fccsw-forum.web.cern.ch/t/solved-port-recent-fccanalyses-compilation-fixes-into-fcceephysicsperformance/107 — Reply to this email directly, view it on GitHub<#40>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABKTDTZJXQ5EQSJKCZUATNDVHKICHANCNFSM5USOA2EA>. You are receiving this because you were mentioned.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub<#40 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABI7YBAQW5BNIVTK7AWJPJLVHKLUXANCNFSM5USOA2EA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[patriziaazzi]

On 28 Apr 2022, at 16:28, EmanuelPerez ***@***.******@***.***>> wrote: There were reasons for setting up things like that - with Clement - more than one year ago. The PhysPerf repository was supposed to contain the analysis-dependent code, things that are very specific to a given analysis and need not be in a central place. This code may not be dependent on FCCAnalysis, but chances are that most of it will be. Also, we thought of it as the place where, besides the documentation, "all what is needed to reproduce a given analysis" would be kept. One could instead have everything within the FCCAnalysis repo - with some subdirectories that separate what is the "core" from what is analysis specific. But before making changes in a rush, one should think whether this will scale since, basically, it means that you will compile your analysis code together with the analysis code of everyone else. Fully agree on the “no rush”, but it is good to start the thought process now we have scaled up and hopefully we will scale even more in the next year. E. On 28 Apr 2022, at 16:08, Patrizia Azzi ***@***.******@***.***>> wrote: Dear Juliette, Indeed what you are asking makes a lot of sense and I thought it would have been the plan all along. In my mind, the FCCPhysicsPerformance directories are a place where people keep the documentation relative to the Case Studies, including some code that is specific and maybe even not so dependent on FCCAnalysis. More like a “working group” space that could have instant documentation via git, and not the main place for the code. Maybe some sort of better “Project management” software could be used for the future once the number of case studies starts increasing considerably… Conversely I had imagined that in FCCAnalysis we could start having a structure as well representing the case studies that would include more “mature” code. So in some sense, indeed, I would merge the more “coding” content of the Performance directory into the FCCAnalysis (to facilitate all these compilation issues in fact), while leaving maybe a similar naming structure so that it could be easier for people to navigate to the code related to a specific case study. I agree this should be discussed urgently. Best, Pat On 28 Apr 2022, at 15:37, Juliette Alimena ***@***.******@***.***>> wrote: @EmanuelPerez<https://github.com/EmanuelPerez> @patriziaazzi<https://github.com/patriziaazzi> We have two repositories, FCCAnalyses and FCCeePhysicsPerformance, and FCCeePhysicsPerformance is very dependent on FCCAnalyses. Seemingly every time that FCCAnalyses is updated, FCCeePhysicsPerformance also needs a corresponding change. This has been an issue that has come up several times now, and it really grinds analysis to a halt as we try to figure out what exactly needs to be synced between the two repositories. Is there a way we could somehow merge the two repositories, or at least make them less dependent on each other? @clementhelsens<https://github.com/clementhelsens> and I have discussed this, and I think we agree this part of the workflow could be significantly improved See also https://fccsw-forum.web.cern.ch/t/solved-port-recent-fccanalyses-compilation-fixes-into-fcceephysicsperformance/107 — Reply to this email directly, view it on GitHub<#40>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABKTDTZJXQ5EQSJKCZUATNDVHKICHANCNFSM5USOA2EA>. You are receiving this because you were mentioned.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub<#40 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABI7YBAQW5BNIVTK7AWJPJLVHKLUXANCNFSM5USOA2EA>. You are receiving this because you were mentioned.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub<#40 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABKTDT43P77XVGAC5QUMHHTVHKOCBANCNFSM5USOA2EA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[vvolkl]

It would be easier avoid the issues that Juliette mentioned by keeping all analyzers in hep-fcc/FCCAnalyses, but there is no fundamental issue with spreading things out over repositories. However, then they should also have the same testing infrastructure we just added to FCCAnalyses, and, in order to keep things consistent, also be built as part of the full stack.

[jalimena]

I think what @vvolkl proposes is excellent.

[clementhelsens]

given the experience we have acquired since one year, I believe that we should now decide what are the best options.
I believe as it was expressed here, that keeping FCCAnalyses analyzers in a same directory makes total sense. In the future, the core of FCCAnalyses will move to k4Analysis which will be some kind of generic RDataFrame wrapper around edm4hep, thus FCCAnalyses will naturally contains case studies specific code. To avoid large compilations, it’s easy to split the repository such that with compilation options we build the relevant ones (granularity to be defined). As a concrete first step I suggest that Juiliette moves the code she needs from FCCeePP to FCCAnalyses such that we start from somewhere (in a dedicated branch)

[jalimena]

Given the recent changes, mostly in HEP-FCC/FCCAnalyses#156, I feel we can close this issue, if everyone agrees. @vvolkl @clementhelsens @EmanuelPerez @patriziaazzi

[clementhelsens]

hello @jalimena , I guess you also want to refer to HEP-FCC/FCCAnalyses#151 ?

[jalimena]

ah yes for sure, thanks @clementhelsens !