Three user stories: How does wallet3 address those? #2
Comments
|
I maintain a payment processor in Golang. It's not possible to link with C++. Currently we have to deploy monero-wallet-rpc server to use wallet functions but it sometimes let's us down when it comes to availability and threading. It is also harder for the user to get monero-wallet-rpc running. What would be ideal for our project is a library that safely implements wallet functions in C. This linking problem isn't unique to Golang. Rust, C and other languages also do not support calling C++ but they do support C. |
|
I go into speculation mode: I speculate you are convinced that it's possible to build what you call here wallet3 in a way that turns all your 3 user stories and possibly many more into a pleasure, because more or less whatever people want to implement "on top" of Monero, those people find that this wallet3 thing just works for them, without much friction, and without the need for much contortions or even hacks. I say that now as only one of multiple members of that "Seraphis wallet workgroup", my personal opinion as a Monero dev: I am pretty sure it's not possible to build something like that. You can't get such wildly differing requirements like you lay them out here with your 3 user stories all under a single hat, if only you are clever and build everything in a certain way. Maybe, just maybe, it would be possible if we could rewrite the whole Monero codebase again from scratch, and if we had the time and the resources to do it, and some world-class devs to pull it through. All things we can't do and don't have. What you wrote so far did not convince me to propose to stop what we are doing in that workgroup, go back to the drawing board, re-group and re-orient. But of course I am only one of the members of that workgroup, not even one with an official title, as you correctly noticed. My personal opinion. Nothing is lost then, this second issue and the first one are open and public, and if you manage to sway the "loose consensus" of the workgroup about what we should build, and how we should build it, no problem whatsoever. I would be ready to also manage a workgroup with a quite different approach, if it still serves the goal of "No wallet left behind", the goal that the hardfork to Seraphis won't decimate the array of software that people can use to manage their XMR. |
|
I am a |
|
On 1:
Today this particular use case is satisfied by a light wallet server. Light wallet servers sync many view only wallets at once, then return data necessary to calculate balance to respective clients. Seraphis/Jamtis enable a light wallet server with stronger privacy protections that similarly would be able to return data necessary to calculate balance to respective clients. On the server: it remains an open question if a light wallet server implementation will be included in the reference monero binaries or shipped/maintained separately as they are today (see monero-lws, openmonero). On the client: @UkoeHB has written wrapper logic to sync and calculate balance using sync'd data returned by a Seraphis compatible light wallet server in his seraphis lib. A wallet3 should be able to use that. |
Implementing a light wallet server is a good example. It is similar to story 1 and 2. The syncing function will be different from what we would typically expect from a mobile wallet for example. It should be possible to sync many wallets at once with the data supplied by one daemon connection. So the API would need to provide a function for processing one block at a time. And there needs to be a clear description of what the syncing algorithm needs to do in the case of reorgs. We need a detailed description of what to look out for there. Or do you think it would be possible to provide a helper function that takes care of the pitfalls and can encapsulate this piece of complexity? Because there are so many different circumstances under which wallet software needs to run, it seems like the syncing functions should not be hard coded in the library. It should be modular enough so that the different wallets can adapt to their use cases. |
@UkoeHB has written helper functions that handle reorgs here: https://github.com/UkoeHB/monero/blob/99b15c2a2c29c7a4163bbbe4512aa451a5edb776/src/seraphis/tx_enote_scanning.cpp
These functions seem to be written with the modularity you want (and I agree that they would ideally have) in mind. Example: notice how |
I am not so sure about this. Here is the part in the presentation where this topic is covered: https://www.youtube.com/watch?v=aAvSpfll9z4&t=6203s It seems like everything is tightly coupled and even networking and concurrency is interwoven into the code. I also sincerely don't understand the aversion to providing a C ABI to this. As mentioned earlier all the other languages can not interface with CPP. So they need to implement custom wrappers and make their own build processes for the Monero codebase. |
All of this is work - work for C++ devs who don't actually like C (at least I don't). You are welcome to propose a rewrite that is pure C. I'd actually be interested to see an alternative to the balance recovery framework currently used, because I don't know a cleaner way to do it that is equally generic - would be great to learn a new way. |
This is not about what you like or not. This is about what is needed: a C ABI. Not providing one means that every single project needs to have a custom build of Monero and create this extern "C" wrapper that you don't want to provide. Here is cake wallet for example: https://github.com/cake-tech/cake_wallet/blob/50fd6accef4cdc63e7f8a172129bfad734110f18/cw_monero/ios/Classes/monero_api.h#L7 |
Sure it is, I have been doing what I like for years and don't plan to stop any time soon :). In any case, if you want a C ABI then by all means go for it. Or perhaps @j-berman in the course of his work will see the bigger picture of how everything fits together into the ecosystem, and will come up with a nice C interface. |
You get paid by this project to solve a problem. This is not your personal pet project "to have fun". I am astonished that you want to move a project with a 2 billion dollar marketcap to a new unproven protocol with this attitude. |
|
Is having fun mutually exclusive with improving a major cryptocurrency? Please, instead of this drama and trying to boss me around (??), go review all the seraphis material and find some specific low-quality thing I produced, some example of me disrespecting Monero core design principles, whatever you like. |
|
@spirobel there are teams of researchers outside of Koe that set the direction of development. This isn't code done on a lark |
Now that I've had some time to work with the lib, I can say with certainty it is written modularly and can be adapted to various use cases. Check out this code I wrote: https://github.com/j-berman/monero-cpp/blob/9f788b40c60d469381d8b9790a0d50aee917f59e/test/scratchpad.cpp#L27-L73 It's using the Seraphis lib to identify a user's spends and receives, reading a mock response to Notice how with the Seraphis lib, there is a scanning util Compare this to wallet2's The Seraphis lib is a clear upgrade. I think it's also worth noting I had an enjoyable experience using and reading the Seraphis lib code to write this code, which is something I generally can't say the same for when I try to do something with or in wallet2. I think the most bothersome thing to me was type conversion, which may have some room for improvement. As far as exposing a C ABI, perhaps some external lib could be written that does something like what Cake Wallet does, exposing functions from within the Seraphis lib via a C wrapper, and includes instructions/examples on how to build the code for use with various languages? Perhaps there is a middle ground on this front compared to rewriting huge sections of the code in C? Bonus: also check out the |
Currently every single wallet and project needs to build these wrappers themselves. Take a look at this for example: which was later increased to 10xmr https://www.reddit.com/r/Monero/comments/zr65zq/bounty_for_libmonero_windows_crosscompile/ It is time for Monero to provide pre compiled libraries with a C ABI so that other projects dont need to write their own custom wrappers anymore! It is not a good idea to spread this out over many different projects! It increases supply chain risk and makes building things with Monero so much harder!
yes the use of threading in this functions makes it a non starter to compile this directly to wasm (which does not have threading) It is a good example of what should be avoided.
great to see monero-cpp (which is used as a basis for monero-javascript.) I use monero-javascript to build an alternative to the wallet-rpc: I had to do this because the wallet-rpc stalls if the daemon connection fails. The concurrency does not work right in this program. It is one of these half abandoned broken tools in the monero core repo. monero-javascript is a step forward. I managed to build a browser wallet similar to metamask on top of it: https://www.youtube.com/watch?v=4DLcsQ45zoE (doing a stagenet transaction in the video that automatically shares the tx_proof with the website after the transaction is done) But because it is still just a monolithic cpp library that is compiled to wasm with emscripten it still crashes sometimes. What we need is either:
It is not that big of a deal to make sure the code has a C ABI. But only for the people that are in charge of the code. if core does not provide this, nobody else can. There is no way to keep up with the ever growing pile of CPP. Especially for outsiders that have no say in the direction of the project.
Do the functions have side effects or shared global state? I saw the monero_wallet_full.cpp still depends heavily on wallet2. There are references to the m_w2 object everywhere. How does the seraphis library avoid this situation? |
|
Sorry I'm taking so long between responses :/
The functions I wrote and referenced above have no side effects nor shared global state. They are purely functional. There is no code path to wallet2; they don't use the I'm working in a contained WIP scratchpad within Identifies enotes (outputs) the user received in a chunk of blocks.
Identifies enotes the user spent and received in a chunk of blocks
The Seraphis lib offers a number of reusable powerful functions that themselves have no side effects nor rely on shared global state. There are 3 main helper scanning utils the Seraphis lib exposes that are doing the heavy lifting in the functions above:
These 3 pure functional scanning utils made my life easier writing this code. They are fairly simple to utilize correctly. The analagous wallet2 functions are:
I think when comparing code side-by-side, it's clear wallet2 does not have nice reusable functions to work with for scanning. And this is because the code wasn't and isn't written with reusability top of mind, whereas the Seraphis lib clearly is/was from the bottom up (across a C++ code base). A path forwardGenerally, I think it could be a good idea to have utility functions a dev can plug and play to scan efficiently in different contexts (plus functions to generate addresses, construct txs, etc.), that are supported and maintained in the core repo, and yea perhaps exported via a C ABI. And caveats such as handling reorgs, lock times, and the burning bug well documented.
I think there have been some misunderstandings/confusion in this thread stemming from the intended scope of your request. To me, it sounds like at the minimum, you would like to see a C wrapper maintained in the core repo like Cake Wallet's, so that people who want to make Monero software don't need some custom solution that does that same thing. At the maximum, you want all core Monero logic to be re-written in C, so that everyone who wants to write complicated Monero software in any language doesn't need to rewrite a lot of code themselves in case they can't find a way to link the C++ (and remove supply chain risk, leaner builds, build complexity, etc.). The former is a reasonable request imo and I don't think it would be a tremendous lift to find a solid way to move forward with something along those lines (this path could also potentially include a lib targeting each platform for devs to use like you suggested). The latter (re-writing everything in C) is a much heftier request that will take lots of high quality motivated dev work to achieve. I think you've made a fairly convincing argument that the former is a reasonable path forward given where things stand. And as mentioned above, I also think it makes sense to shoot for utility functions exported via a C ABI that can be plug and played to accomplish your user stories. I think it would be nice to settle on a more grounded path forward here. That being said...
I empathize with your pain here. But, I think it's also worth keeping in mind you're requesting people support more in the core repo by maintaining a C ABI. The wider in scope the core repo becomes, I think the more difficult it will be to maintain into perpetuity to avoid these sorts of complaints. I'm thinking this ABI could strictly export functions that are used internally in the core repo's tools that the core repo is responsible for maintaining (e.g. CLI wallet, RPC wallet). Thoughts? On the other wallet-rpc comments:
This sounds like a bug. If you can repro it would be good to submit an issue in the main monero repo
Sounds like you're running into another bug here that would also be good to report. This sounds like a pretty interesting issue |
|
Thank you very much for this comment! This is exactly what I mean! Maybe this does not sound like a big deal, but a C ABI will make a big difference for Monero adoption! If we want to see new things built with Monero, we need to make it easy and pleasant to work with. Btw here is another wrapper that has to duplicate the work that could be done all at once in the core repo: https://github.com/mollyim/monero-wallet-sdk
Maintaining this ABI in the core repo will free up resources across the ecosystem. We should get the different stakeholders ( wallet developers of all the wallet projects) involved with this so we can see where the pain points are and what could be improved. This might even lead to a reduction in scope, because we can identify the APIs that get the most use in practice and focus on maintaining those. It will also make it easier for the "no wallet left behind" initiative by @rbrunner7 to get everyone on board, if we make it an explicit goal to provide a C ABI by the core repo. It does not mean that we need to rewrite everything in C. We can start slowly. |
Certainly a worthy goal
I think there's a higher likelihood for success with this route if stakeholders are approached with a concrete base of a proposal, and then collaboratively build from that base. How does this sequence of steps sound:
Once you've got a number of existing stakeholders demonstrating positive feedback for this, it'll be easier to move forwards with that base, flesh out scope, harden an API, and get more people on board, etc. etc. My main hesitation here: maintaining something in the core repo that isn't used in tools the core repo is responsible for may prove challenging into perpetuity. I generally think a concerted effort toward improving Monero's tooling and developer experience is well worth it, but it's important to be realistic about what will be maintained in the core repo (at high quality) into perpetuity. Perhaps if a number of stakeholders want to see this ABI, and would prefer to use something like it in their apps, then it would make sense to e.g. re-wire the GUI to use this ABI as a way to make sure the core repo maintains it. |
|
Very interesting thoughts, @j-berman. And quite in general I like how this discussion now takes a positive turn compared with some ... not so positive things ... that were exchanged earlier in the discussion. The argument to save work over the whole ecosystem is certainly valid, but should it turn out that the devs working on the core software just can't help themselves because they run on their last legs it gets of course difficult. And I think to remember to have seen some wonders of open source and the sharing of work that it enables: Several wallets seems to use code from Monerujo for their integration, thus at least some duplication of work was avoided. |
the seraphis lib can also handle legacy transactions and outputs, right? From what I have gathered also reading the writings by @rbrunner7 on seraphis migration strategy concerns: It seems to be non trival to get all the stakeholders on board for a big change like this.
There is still a need to have custom builds though. That adds complication and complication adds supply chain risks. So we could get people to migrate now instead of waiting until the hardfork moves closer. Because it will save time and hassle immediately. (and it will also future proof their efforts)
The issue is that GUI is a cpp QT app so it does not face the issue of not being able to interoperate with cpp code. What we need is either: A) a non cpp wallet in the core repo. or B) make core feel responsible for the whole community and not just the stuff that gets checked into core.
seems like a plan. A related question that I am interested in: I want to use this C ABI with zig as a build system to compile it directly to wasi. I also wonder if others are interested in wasm and especially wasi builds. |
Yep :) I used the scanner implemented above to successfully scan a mainnet wallet. Still hardening it/testing older tx versions.
I like this general strategy. I would think stakeholders would respond favorably to the idea they'd receive a library that handles their Monero core needs in potentially cleaner/simpler/safer fashion than their current method of using core Monero code + also handles the Seraphis upgrade.
"A" seems unrealistic to me as it widens scope of the core repo a significant amount. B is already the case generally, e.g. see: monero-project/monero#6332. As it relates to this specific discussion (a C ABI in the core repo), it primarily requires willing maintainers and contributors to the C ABI into perpetuity. With enough demand for it, I think this would come naturally, but until that demand is demonstrated and proven from stakeholders, then it's hard to see this route succeeding. I understand that the GUI wouldn't necessarily need to go through a C ABI, but I don't see why it couldn't interoperate with the core repo code via the ABI, thus establishing the structure where core repo maintainers would also need to think about how the ABI is used by wallet devs by default. In any case, if a large number of wallet devs in the community really want a C ABI in the core repo, then I think it will get maintainers/contributors naturally and it would be superfluous to re-wire the GUI through it. So next step for this general strategy to me seems like we should build a base, and approach stakeholders with that base to get feedback and gauge demand. Would you be willing to build this base and put the structure of ideas you've been thinking into code :) |
I am willing to. But I am still hedging my bets. It seems like the seraphis curve might get changed and that means we might have to introduce rust into the toolchain: So am I still on the fence of building on the cpp lib with a C ABI or moving over to rust completely. I started reading the serai monero library. And I am starting to think it is better to wait and see how things play out and maybe we get the chance to ditch the cpp toolchain entirely. I managed to compile parts of the Monero codebase with zig as a cross compiler. So it would be viable to do the same thing with the seraphis library. (also building on your great explanations!) My ultimate goal is a direct build to wasi. The C ABI is a side effect of this goal. But there is also the path through rust. Their toolchain also supports a wasi target. Which path is the better one ultimately depends on how things develop regarding the curve switch. But there is also not just me. I heard the molly devs put a lot of work into this as part of their CCS. So I think its good to communicate first without building a base. It seems like a common problem in Monero that people build their own bases everywhere instead of working together. |
|
Molly.IM developer details their struggles in needing to create a monero-wallet-sdk for android to handle their requirements |
|
@sneurlax do you have an opinion on this monero wallet sdk for android? https://github.com/mollyim/monero-wallet-sdk in terms of it being "modular" and "easy to port" to a 'wallet3' or how it would compare to stackwallet? |
|
monero_c
monero.dart
|
Have you tried using the recent https://github.com/MrCyjaneK/monero_c, @sausagenoods ? |
Hi plowsoff, sorry for losing track of this issue for over a year :) Referencing both links shared re: this issue,
Monero's C++ is more reviewed than some new, inherently-unreviewed tool imo. Perhaps that's an uninformed opinion of mine tho, I'm aware of Monero's review's, but not Molly's.
the monero_c mentioned by jermanuts above achieved this
This is a legitimate concern and leads to wallet binaries being detected as malware due to RandomX hashing and mining-related code in the binaries. I'm glad to see that the CCS proposal was funded and produced this tool: https://github.com/mollyim/monero-wallet-sdk This tool may be usable cross-platform, but I'm not sure it's useful for cross-platform use. It can probably be ported to iOS and other platforms, but I'm not sure: I'm not a Kotlin coder. So my opinions on this topic, I think, are not very important. I'm going to be looking at this later in terms of what API they selected, because lately I've been concerned with what API to use for new wallet tools in general |
The exchange backend is written in golang and needs to sync many wallets at once. The wallets are view only. The goal is to fetch the blocks only once from the node (that is hosted on a different server) and use the data to sync many different wallets in parallel. The Monero related code should be directly integrated into the go binaries for ease of deployment and maintenance.
The team decided to target wasm because it means the code will be platform independent. In this way they can rest assured their code will run on many different operating systems like Mac, Windows and Linux. So even with a small team they are able to service a wide range of users on many different systems. They can focus most of their development efforts on crafting a great UX, because they dont run into many build issues and platform specific bugs. Their protocol also requires the simultaneous syncing of many public view only wallets.
While wasm is a single threaded target without garbage collection (the memory can only grow). They never run into memory overflow issues because they have very small pieces of wasm that get orchestrated by a javascript codebase (that uses webworkers for threading and fetch to retrieve the blocks from the daemon)
A widely used mobile wallet that focuses on reliability and an easy to use user interface. Many users depend on their software so the most important goal is having a robust architecture. They take extra care to make sure there are no concurrency bugs in the wallet. They don't want the user interface to ever become unresponsive under any circumstances. Even if there is a connection loss or the power saving features of the mobile operating system kick in: the user experience should not be degraded.
How does wallet3 address these user stories?
Which API calls would the respective developer use?
How can all of these different use cases with different targets and languages be addressed without a C ABI?
The text was updated successfully, but these errors were encountered: