-
Notifications
You must be signed in to change notification settings - Fork 125
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add comparison entry in "JuliaPackageComparisons" #884
Comments
Great initiative! I feel like the agents page may become the first one with proper content, produced in the way I am dreaming of 👏 I have unfortunately never touched agents based modeling, so I can contribute nothing technical. But I can absolutely comment on how easy it is to understand as an outsider, and general style and format of the comparison. |
I think we can even tag the people developing https://github.com/dsb-lab/CellBasedModels.jl/ and https://github.com/mastrof/MicrobeAgents.jl, these two packages are still not in the general registry and in a early stage of development though, but they seem very good |
In my case, MicrobeAgents.jl it's not really a distinct package, just an extension of Agents.jl for some specific type of problems. I guess a short description would be along the lines of (I will provide something better): |
Great idea. Here my contribution for Vahana: Vahana.jl is an ABM framework tailored for the development of large-scale agent-based models, based on a synchronous graph dynamical system approach. One of its notable strengths lies in its parallel execution capabilities, making it suitable for supercomputer clusters and for handling large datasets or agent populations (an aspect for which there are few alternatives even outside the Julia community). Therefore, a major focus of Vahana's development has been on CPU performance and a small memory footprint. Furthermore, Vahana is optimized for representing complex network structures, making it a good choice for network-centric models. On the downside, while Vahana supports spatial information, it is limited to discrete n-dimensional rasters, and it is less suitable for simulations where agents need to move extensively within these spaces. The requirement to express models as graphs is unconventional and may require a paradigm shift for those accustomed to more traditional agent-based modeling approaches. In addition, some ABMs, e.g., such as those where only a single agent can occupy a cell in a grid and can move (as in the Schelling model), are not easy to formulate in a parallel version at all. There exist also an ABM package with the name EasyABM.jl, maybe @RenuSolanki can add some sentences about it. |
EasyABM.jl is suspiciously similar to Agents.jl with a dramatically smaller range of features. It doesn't seem "easier" to use despite its name, as the usage seems identical to Agents.jl. Perhaps the authors can convince me otherwise. |
Dear @Datseris ,
The code of EasyABM is open source under MIT license. If you doubt it's
similar to Agents.jl, which it of course is not, you can go through the
code and check for yourself. As far as features is concerned EasyABM is a
work in progress and being updated regularly.
Regards,
Renu
…On Sat, 23 Sep, 2023, 2:54 am George Datseris, ***@***.***> wrote:
EasyABM.jl is suspiciously similar to Agents.jl with a dramatically
smaller range of features. It doesn't seem "easier" to use despite its
name, as the usage seems identical to Agents.jl. Perhaps the authors can
convince me otherwise.
—
Reply to this email directly, view it on GitHub
<#884 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AZFWBJQDY56JCJMH4STLDBTX3X6XHANCNFSM6AAAAAA46O5L3U>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Hi @s-fuerst, Anyways, if you are interested in parallel implementation of Schelling like models using Vahana.jl there is some existing research you can refer to. E.g. https://www.academia.edu/35362094/Design_and_implementation_strategy_of_a_parallel_agent_based_Schelling_model |
A bit off topic, but next time you think something is "untuitive" in a Julia package, please open an issue at that package to express your concerns. This helps the Julia community. Helps the devs see problems with their package, helps the existing users get more simplicity, and helps new users gets more accessibility. At least now that I saw some concerns, I will try to improve the documentation of Agents.jl to make it e.g., more clear that you don't need an agent stepping function. For this topic in particular however, and in contrast to the other aforementioned packages, it is difficult for me to create a summary text of situations to use EasyABM.jl in. Here we are drafting a comparison that would help the Julia community choose the correct tool, hence we need arguments that are based on code and data instead of opinions of what is easier or not. Hence, if we want to write "package X is easier in doing Y than package Z", we need to have good reasons to do so. It would be great if you can address the following concerns I have following your arguments:
|
Here I leave a more detailed description of CellBasedModels.jl to help making the comparison. Cell based models is an agent based package that has in mind its application to continuum space ABMs. Its main target is to solve physical models that are, at least partially, defined in terms of differential equations.
NOTE: (I feel that, although Agents.jl claims that can use DifferentialEquations.jl in their models, the example provided does not really show a real compatibility. The integrations used by DifferentialEquations.jl are only performed between agent steps, and the integrators are not used to update the agent steps itself, which is something you want to do when defining ABMs with DE. From the code structure, I cannot come up with a simple fix to this or come around. But this is a problem of a different topic). |
yes, I agree with your note on DifferentialEquations.jl usage. But I think this is what an "integration" would mean: that you may bring in functionality from other packages without disrupting your workflow. In CellBasedModels.jl, we have something much more than integration. DiffEq is a part of the internal package infrastructure. (Although I am sure some functionality like that could be added in Agents.j in the future) |
Just to add a quick appreciation for the usefulness of these comparisons: I had no clue about the existence of CellBasedModels.jl but now I'm wondering if - at some point in the future - there could be room to extend it in a way that is similar to what I'm doing with MicrobeAgents.jl. |
@Datseris mutable struct SchellingAgent <: AbstractAgent What the heck is mutable struct ... AbstractAgent ?. This gave me headaches when I was still learning Julia and knew nothing more than functions, loops and conditional statements. Also, its not possible to redefine a struct in a notebook environment and the user has to reboot kernel if they need to modify properties of agents. Anyways, let me point out some notable features in EasyABM that (AFAIK) are not available in the Agents.jl package, and which add to its enhanced user-friendliness:
More features may be added in future versions. Though we will prefer to keep it simple and lightweight. |
Yes, I agree @Datseris. I just mention it to make clear the difference between our packages and reinforce the specific target of with my package, which is somewhere between ABM and particle simulation dynamics. |
Here is a short description of MEDYAN.jl Mechanochemical Dynamics of Active Networks (MEDYAN) is an efficient and scalable computational model for mechanochemical simulations of active matter networks. Our goal is to be able to simulate both the mechanics and chemistry of the cytoskeleton in a whole cell at the minute time scale while still keeping track of the stochastic chemistry of individual proteins. MEDYAN.jl is not released yet and is still a work in progress but there are currently some basic tutorials here: https://medyan.org/julia-docs/dev/tutorials/1install.html |
Thanks, @nhz2 . When released, will MEDYAN.jl be on GitHub? At the moment it does not have a github repo. |
When released, MEDYAN.jl's source code will be downloadable for scientific use, but it probably won't be on GitHub or in the general registry. I am also trying to separate out the components of MEDYAN.jl that might be useful as stand-alone packages and publish those to GitHub here https://github.com/medyan-dev. |
Interesting, what is the reason for this development approach, versus the typical "do everything on GitHub" and publish on the general registry? |
Unfortunately, the University of Maryland wants us to not have everything on a public git repo (at least for now). In terms of publishing to the general registry, I'm not sure if the University of Maryland will accept any of the OSI-approved licenses. |
To see where EasyABM.jl stands in terms of performance in comparison of Agents.jl I performed that simple benchmark: I ran the Schelling model in a small and large configuration in both framework. You can find the benchmark and its results in this gist: https://gist.github.com/Tortar/90607e8d05dbd1630ad817d8a1b649aa As you can see, Agents.jl is roughly 25x faster in the small benchmark and 60x faster in the large one, while also allocating much less than EasyABM.jl . So the performance of EasyABM.jl is not great. Note also that in this example some choices made in EasyABM like space as “patches” (more on this later) are not benchmarked so in more complex models the difference could probably be even higher. That said, I need to say that I agree with @Datseris when he says that EasyABM.jl doesn't seem easier to use than Agents.jl. In particular I'd like to respond to the points @RenuSolanki made about the strengths of EasyABM.jl:
Is this really an advantage? In our ABM framework comparison we have shown again and again that the approach of "patches" of NetLogo leads to poor performance. It also leads to lengthier code, as it doesn't allow itself to the extremely simple expressivity of Arrays. Any spatial property can be naturally expressed as an array over the space without the need of additional concepts like patches.
Graphs.jl is an exceptional package with a huge list of features. I'm not sure why re-inventing the wheel and making usage of graphs "without external packages", and hence disallowing access to all algorithms of Graphs.jl, should be considered an advantage. Also notice that also in Agents.jl, when a GraphSpace is employed, adding or removing nodes and edges during the simulation is possible.
Actually Agents.jl can use any Makie backend and between them there is WGLMakie.jl which uses WebGL as EasyABM does so you don’t actually need an OpenGL graphics card
First, for sure the majority would agree that configuring the dynamics in the dynamics part, and the visualization in the visualization part, makes more sense than the other way. But regardless, there are two flawed arguments presented here. The first is that "due to this choice" the "animations happen in one line of code". Well, since the rest 20 l.o.c. of animation happen in the model evolution instead... :D Second, the argument that "NetLogo does it this way so it's a good way" is not convincing. NetLogo does many things very suboptimally, which is why its performance is much worse than Agents.jl as proved in https://github.com/JuliaDynamics/ABM_Framework_Comparisons.
You can assign new properties to everything by adding a field "extras::Dict{Symbol, Any}" in agents. It is not useful to do this by default because it is abysmal for performance, so no one should do it, and that is why we don't motivate it. But it is trivial to do, and doesn’t need any “additional infrastructure” or “additional source code” in Agents.jl.
We have this feature for multi-agent models to record different properties in relation to the different types, it hard to tell what this would be useful for in other situations, just because something is possible doesn't mean it is useful. Many of the alternative ways to do things in EasyABM.jl were actually considered at some point in the development of Agents.jl. They were not implemented because they are bad for performance, and the same results can be achieved with different and more performant methodologies. All in all, while I acknowledge that the decisions taken in EasyABM.jl are different, they don't seem easier to reason about and they incur a huge performance drop for the users. |
I've already pointed out in other comments that EasyABM wasn't created to break any performance records. EasyABM is specifically designed for learning about Agents based models and enabling rapid implementation of small-scale ABMs into code. In any case, for a rigorous exploration of an Agent-Based Model most people, including myself, prefer to write the code from scratch rather than relying on an ABM package. This way, there's no room for ambiguity, and one can be crystal clear about every aspect of the model's inner workings. Furthermore, I fully acknowledge that what is easy for one individual may appear difficult to another. There will be users who will find EasyABM to be not so user-friendly. In my personal experience, I found simulating agents based models with Agents.jl to be extremely confusing, which led me to create EasyABM. Dissatisfaction serves as a driving force in open source software development. Therefore, instead of insisting that Agents.jl has absolutely no shortcomings and is "universally easy" to use, I urge you guys to recognise that while it may be a great package and may be easy and user-friendly for some individuals, it may not be so for others, including myself and potentially many others in our community. PS - mutable struct SchellingAgent <: AbstractAgent LOL! |
Noone ever said that. If that was the case, stuff like Vanaha.jl wouldn't exist, which actually counters more than one shortcomings of Agents.jl. Besides, there are 50+ open issues in Agents.jl repo listing many shortcomings. Would we make this information public if we were delusional that Agents.jl has no shortcomings as you seem to suggest? Even more interestingly, why would we want to promote a listing of other ABM packages if we truly believed that Agents.jl is perfect?
No-one said this either. You provided some arguments and @Tortar responded showing how many of the claims you made about "EasyABM is easier than Agents.jl" (your words), don't stand up to scrutiny or make sense from the perspective of a typical Julia user. I don't see where the "universally" comes into play. To re-iterate the purpose of this thread: it is to provide the Julia community with objective arguments on why they should be using one of the many possibilities for Agent Based Modelling in Julia. Another developer argued that EasyABM.jl should be in this comparison. So, can you give us a summary of what EasyABM.jl brings to the table from an objective point of view? Is its goal to be used by a community of users that do not use the Julia language and hence cannot comprehend concepts like |
You still don't get it. I find EasyABM easier. I find Agents.jl difficult. Many users including my collaborators feel the same. There is nothing to prove or disprove here.
I did. This was followed by Tortar's argument that (with some hacks) Agents.jl can also do the same. Both being ABM packages it was not a surprise for me that both packages have similar capabilities.
Yes it is. Many users of ABM packages come from backgrounds like sociology / biology / ecology / economy and face immense difficulty understanding concepts like structs and types. |
I guess that it will not always be possible to reach 100% agreement on what the strengths and weaknesses of each package is. I want to mention that I am not opposed to presenting opposing views in the comparison. E.g. "Some feel that PackageA is the easiest to use due to the convenience-functions it provides. Others have reported that they find the way PackageB composes to result in larger models being much simpler to define". But also, keep in mind that the users will not be the developers, and the comparisons should be focused on helping users. So the opinions of external users is more valuable/relevant than the opinions of the main developer, especially if there is essentially a single developer (as is the case for some projects). The main developer(s) can speak to the goal of a project, but outside users know best what it is like to use the package. I think it would be productive to think a little deeply about why you find one package easier to use than the other, and to keep in mind that you will be biased toward the one you work on because you understand it much better. List concrete reasons and give examples. That will really help paint the picture in a comparison, especially if the picture is not clear-cut. |
hi guys, allow me to chime in real quick. A sociologist here, I know pretty much nothing about Julia (but I am learning), I am a NetLogo good user. It took me an afternoon to clone my model with Agent.jl, run time went from 2 mins in NetLogo to 19 secs, also the number of lines of code went down considerably... I am amazed by the Agents.jl package! IMHO, the real difference for the user is made by the documentation: easy means thoroughly explained, also with example that tells you the obvious thing (like if you launch a distributed batch of simulation then you need to explicitly remove the workers when you're done - I crashed my server... :-) only thing that really is a bummer is that with notebooks on VSCode every time you are forced to restart because you modify some immutable structure, then I end up with a stuck Julia process... |
thanks for writing about your experience @manentai! Indeed the problem with redefining new types is a long-standing limitation of Julia itself, see https://discourse.julialang.org/t/redefining-structs-without-restart/50826 or timholy/Revise.jl#18. There is a workaround described here: https://discourse.julialang.org/t/redefining-structs-without-restart/50826/8 but I don't know if it is applicable in all situations. So it is difficult/impossible to find a "trick" for Agents.jl which keeps the performance and the simplicity at the same time while also allowing redefinitions. We can hope that Julia devs will be able to lift this limitation in the future :-) |
I think it is time to close this. We are only missing a summary for "EasyABM.jl" which I will provide at the end. First things first, let's clarify somethings Using structs is not difficult for the overwhelming majority of scientists, irrespectively of background. I have given invited talks about Agents.jl in the following places:
None of them found weird the struct approach Agents.jl or difficult to get started with. That's my proof that this concept isn't complex or advanced, and it isn't my opinion: the list of people I have taught and interacted with is diverse enough. Therefore even if @RenuSolanki does find structs a complex concept, this applies only to themselves and their collaborators unless more objective data are provided. Personally, I find the name "EasyABM.jl" insulting as it implies that the rest of the ABM software are not easy in some way which is simply not true. Besides, whether a software is "easy" or not depends much more on the documentation of the package rather than the implementations in the source code. That is why I requested a package rename: RenuSolanki/EasyABM.jl#63
The only difference that actually exists so far is that EasyABM.jl doesn't require users to define a custom struct. All the other things it does, you can do in Agents.jl, or in any other framework here as far as I can tell. Oddly, when I go to the API page of the documentation of EasyABM.jl the very first thing is that it actually uses agent types: What is
Agents.jl and EasyABM.jl don't have similar capabilities. By any metric you can come up with, the list of features of Agents.jl is at least 10-100x the list of features of EasyABM.jl. With these comments out of the way, the only summary I can think of for EasyABM.jl that is just to the other packages is: """ Anything beyond that would need to be justified with objective facts and not opinions. |
I find this summary too harsh and negative. And it ignores other differences that RenuSolanki mentioned, e.g. that it's closer to the NetLogo philosophy (which I don't like either, btw). I have now compared the Schelling implementation of both frameworks and actually find EasyABM's easier to read. Just one example of a few that caught my eye, namely the controls of the interactive app. In EasyABM we have these directly as parameters with an easily understandable parameter name:
In Agents.jl we create an interactive app with the
When I look at the API documentation, I'm totally confused, and would say that you can't understand how to add controls to an interactive app without having seen an example in a tutorial. In EasyABM create_interactive_app is actually not documented at all ("Creates an interactive app for the model."), but a click on the "Source" button is enough to directly see all the kwargs and understand how it works. I understand your annoyance at the name, and would certainly go for Agents.jl rather than EasyABM.jl, but my suggestion for the summary would be this one: """ EasyABM.jl is general purpose ABM framework with a philosophy that is more oriented towards NetLogo and tries to offer a simple approach to ABMs for people without experience in Julia, at the expense of performance and flexibility. """ People who have as a high priority that the framework should be easy to learn hopefully still take a look at Agents.jl and then decide for themselves which framework is more suitable for them. However, I think it would be bold to claim that Agents.jl is superior in all cases. |
Great, thank you for your reply. A third eye view is clearly what was necessary. Sure, I agree with using your summary instead, even if personally I do not agree with the "simple" claim. |
Hi Renu, you know what, you are totally right. I've been too negative, I am sorry. I could pretend its because I've spent my Xmas in the hospital and I am overall full of negativity at the moment, but that isn't any excuse as I can see I have overall been negative in this issue, so sorry about that. So I take your positive spirit and to heart and say the same, wishing every team here a happy new year :) |
I've created the first draft of the comparison page here that anyone can comment on if they feel I wrote things they disagree with, such as e.g., the summary for Agents.jl! https://github.com/JuliaPackageComparisons/JuliaPackageComparisons.github.io/pull/34\ It is still a draft as I haven\t mentioned yet Agents.jl extensions such as the demographic or microbe models! |
I'm very sorry to learn about your health @Datseris. I also apologise for the negative tone in my comments. In fact, reflecting on this year, I've been involved in numerous online arguments on social platforms :P – a trend I'd like to avoid in the upcoming year. Take care, and I hope you feel better soon 🥳 🍀🌈. |
Oh wow, I am super impressed at how you managed to connect across the severe disagreements that arose along the way. Thank you everyone for bringing your effort and energy into this discussion, and for helping making the comparison page more representative. Sidenote: I am very much looking for others to help maintain and develop JuliaPackageComparisons, so do not hesitate to reach out if you want to become a developer of it! (I.e. get merge rights to the repo) |
the project https://github.com/JuliaPackageComparisons/juliapackagecomparisons.github.io
is something the julia community and especially newcomers can benefit dramatically from. Now with the advent of more packages for agent based modelling in Julia, we should coordinate and write a page there. We can draft the page here!
I am cc'ing other people who would likely want their package to be in the comparison, so that they may keep track of this issue and approve the final text that we may submit as a PR to the comparison packages!
If the authors could provide a couple of sentences of the pros and cons (when to use it, when not), that would be fantastic.
p.s.: @KronosTheLate thanks for the effort coordinate the comparison website. I think it will be fantastic for newcomers!
The text was updated successfully, but these errors were encountered: