FATAL: Caught exception: It appears that Fulcrum was forcefully killed in the middle of committing a block... #41
Comments
|
Yeah it's a known issue with the way I did the data layout. I will have to redesign the data layout to avoid this in a future version. The recommended way to stop Fulcrum is to send it You will have to resynch, unfortunately. :/ Sorry about that. A future version will try to be ACID -- but for now I took speed shortcuts -- so hard shutdown runs the risk of this issue happening if you shut down in the middle of when a block arrived and the DB was being updated. I understand that ElectrumX did not suffer from this. It was also slower. :) I will see if I can do ACID without too much of a perf. hit in a future version. For now you will have to resynch from scratch though. Sorry... If this makes you worried you can always also backup the synched DB (with Fulcrum stopped). That way you can always restore from backup. FWIW I have been running my server for months now and never had to restore from backup. Sorry about that. |
|
Awesome! Thanks for the quick reply.
Would you be able to give me a hint on where to look so that I can implement it? The problem is not that I force-kill a process, the problem is the server could be under highroad and the OS terminates the process abruptly. I'm curious about the locking here: https://github.com/cculianu/Fulcrum/blob/master/src/Storage.cpp#L1232 Does this mean that while we are processing a block, that no one can query mempool/utxo's (ie: blocked threads) until the block is committed? Thanks so much! |
What's in your systemd file? I don't have RestartSec=60s in it, but it never crashed. |
|
@atteeela --
It's not a trivial fix. It would require redesigning the database to use a single table with different "column families" for each of the logical pieces: utxo_set, scripthash_history, headers, etc. If all of the data lives in a single table it's possible to do "begin transaction", "end transaction" pairs when updating the data when new blocks arrive, and it would be ACID -- in that case even yanking the power chord or whatever will not lead to any corruption (just a rollback). So.. given that, it's more than a quick hackjob -- a person with significant experience in rocksdb and C++ would be needed to do this. I can do this myself -- and the reason why I didn't do it initially was because it was slower than the data layout I have now. I wanted to design this server to be as fast as possible. Potentially the new data layout would be optional and only for users that prefer ACID over speed. So -- the database layer would need to be abstracted a bit to handle both data layouts. It's not a small job; I can do it myself -- it just would take me a lot of time.
Yes, that's correct. Typically the locks are held for less than 5ms (sometimes less than 1ms). And blocks arrive once every 10 minutes. It's not exactly a huge deal. This amount to: 0.00083% of the time that locks are being held exclusively, on average. You may get more of a burp and slowdown from your OS kernel than from this. The rest of the time everything is very much parallel. |
|
I'm running into this issue. It's annoying that I have to start sync over from the beginning. I tried tweaking the db_max_open_files, max_pending_connections, and bitcoind_throttle values down but it's still happening around block 450,000. Can anyone recommend a workaround? Thanks |
|
@vul-ture Is it happening on initial synch? Or later? Don't kill Fulcrum with |
|
it happens on initial sync, I'm not even sending a signal to fulcrum. It could be that my RPC connections are getting saturated (? |
|
Huh? If you haven't synched yet RPC is not even up yet. There are two possibilities here:
Please ensure that the directory you use is on a filesystem that has ~40GB free for mainnet. |
|
Also please ensure you are using a filesystem that supports >2GB files... (e.g. no FAT32 or other ancient filesystem). |
|
I meant the RPC connections to bitcoin daemon |
|
RPC to bitcoind being too slow shouldn't lead to this error message -- you would instead see some warnings about connections dropped / reconnect to bitcoind -- but it would recover from that. |
|
Right, the daemon connection is fast as well. It must be DB write issue. Performance of this app is excellent, ~10k transactions/sec. If I can work around this error and sync it should work fine. My specs are pretty good so I don't think it's a hardware issue. |
|
Ok well without more info I can't help. I still think somehow your data dir is not on a filesystem with enough space ... Maybe some verbose logging will elucidate things, one might hope. |
|
OK I'm able to reproduce the problem, this might be a different issue than the original bug. [2020-10-04 12:19:27.431] Verifying headers ... console reports Looks like my bitcoind might have a corrupted block? investigating |
|
Oh man! You’re on bitcoin core (btc). Fulcrum is for Bitcoin Cash. However you piqued my curiosity. I will try and get it working with bitcoin btc (maybe) in the coming week or two. Yeah fulcrum is for bch... That explains it! |
|
Lol oops, sorry about that! I will use it for BCH then |
|
Hey @vul-ture if you want you can try using Fulcrum with BTC if you start I am going to test this myself but from my reading of the bitcoin core sourcecode it should work. |
|
testing now with the rpcserialversion flag |
|
Hey! @vul-ture So actually latest Fulcrum release fully supports BTC. Don't use that flag -- it will give you problems later when you connect Electrum to it. Just get rid of that flag, blow away your existing Fulcrum db, and resynch again from scratch. When it finishes synching you can serve up BTC. I have been doing so for the past week without problems. I have 222 users connected right now. |
|
Working, thanks! I think this is the fastest electrum server I've seen. |
|
Wow man thanks for the compliment. :) Yeah I tried to make it fast FAST. That was my #1 design goal. (And of course also correctness was too). Thanks! |
|
@vul-ture PS: Be sure to be on latest Fulcrum 1.4.0 since in the 1.3.x series BTC support was still beta and the mempool code wasn't as fast as it is in 1.4.x series... If you aren't already on Fulcrum 1.4.x, upgrading is simple and doesn't require a db resynch. |
|
Just tagging a comment to say I ran into this issue too. I had to forcibly kill my VM where Fulcrum was running on. Then when I brought the VM back up I got this error. |
|
@apemithrandir I see. Sorry to hear that. I do plan on making Fulcrum more resilient to unfortunately-timed crashes in the future. It is a partially solvable problem, meaning that we can get it to a state where for 99% of crashes it should be able to auto-recover. It just requires more logic in the code to diagnose what went wrong and backtrack a bit. Thanks for the feedback. I have never had to re-sync fulcrum after hard system reset and I've had my server randomly lose power or be forced to randomly reset about 12 times in the last 3 years (I was running my server out of my apartment at one point, without battery backup, ha ha). I sort of optimistically thought that such corruption issues would be rare. But this just proves that anything that can go wrong will, eventually, given a large enough install base. |
|
I think it was a combination of my VM being off for a few days and I was bringing it back online. Bitcoind was still grabbing blocks and fulcrum was also still updating. My VM was acting laggy/unresponsive and then I went to restart. When restarting the VM, it just hung on a black screen and I had to do a force turn off. |
|
Although it pains me to say it, I would give up a little of Fulcrum's stunning performance (if necessary) to have this implemented. Although it tends to happen more during initial indexing (often due to an overly optimistic |
|
Yeah, it only happens if Fulcrum was in the process of writing out data for a new block -- so during a catch-up phase or a sync it can happen after a non-graceful exit. Under normal operation it's unlikely since a block arrives once every 10 mins and only takes 20-100msec to process, depending on CPU and HD speed.. But yes, this is solvable and I will focus on that in the future. 100% agreed. |
|
I am struggling to get through the re-sync without hitting this error. I'm on my 3rd attempt now. This is the most recent forceful kill log:
I was using this in fulcrum.conf:
Any suggestions? |
|
Maybe it is a problem with my VM, but twice now when I rebooted my machine after Fulcrum failed I have been booted into a initramfs prompts/busybox screen and had to manually perform fsck command to recover my file system. |
|
I am really sorry this is happening. So there must be a bug in either the The strange thing is that I have synched recently on BTC just to test things out and I never observed this behavior. So it may be something specific that triggers it. Yes, the more I think about it.. it could be that |
|
Follow-up: I predict without fast-sync it won't fail. This would be evidence that |
|
Ok. I assume commenting out the fast-sync line in my fulcrum.conf is how I run without fast-sync. I did run into my CPU maxing out during the last run I did, so I've set the worker_threads back to 1 from 2. |
|
After over 2 days on the Sync (with fast sync disabled) at block height 653,000 my CPU locked up again. Since I had the worker_threads=1, the CPU locked at < 100%. The VM was still unresponsive though. |
|
Sorry to say, I was unable to get Fulcrum up and fully sync'd after a week of trying. My VM kept crashing or freezing during the re-sync, forcing me to do another re-sync. |
|
I'm sorry to hear that, @apemithrandir . I have had little trouble synching it even on old Windows 7 boxes (yes, there is a windows .exe available) with like 4GB of RAM and HDD. It's perplexing that it would fail on what sounds like more generous hardware. I'm just curious -- can you provide more details about your setup? Like host os, guest os, VM software, VM configuration, host machine specs, and relevant parts of config file, etc. Anything helps. I want to see if I can reproduce the issues you experienced. Sorry to hear you are going :(. What about running on bare metal outside a VM? Or using Docker? |
Happy to share any and all details with you one on one over private message/email, if it might help you with development. |
I just experienced the same thing. My VM rebooted for updates. [2022-05-21 20:06:24.266] Coin: BTC
[2022-05-21 20:06:24.266] Chain: main
[2022-05-21 20:06:24.267] FATAL: Caught exception: It appears that Fulcrum was forcefully killed in the middle of committing a block to the db. We cannot figure out where exactly in the update process Fulcrum was killed, so we cannot undo the inconsistent state caused by the unexpected shutdown. Sorry!
The database has been corrupted. Please delete the datadir and resynch to bitcoind.
[2022-05-21 20:06:24.268] Stopping Controller ...
[2022-05-21 20:06:24.268] Closing storage ...
[2022-05-21 20:06:24.341] Shutdown complete |
|
Yes, I'm sorry. This happens if it's killed while it's busy processing a block. Perhaps you should setup fulcrum to be a systemd service this way any reboot of the node will send it a SIGTERM or whatever to shutdown gracefully? I'm sorry. You must resynch now... |
I'm running it inside of a docker container, so I'll have to figure out a graceful exit strategy. I appreciate the reply. |
|
Yeah, I'm sorry. I will have to redo the data model to be fully ACID and then this can never happen. This is on the to-do list for Fulcrum 2.0. Sorry about that. |
Looking forward to it. Thanks for prioritizing it in the project. I'd offer to help, but I mostly code in python. |
|
Just wanted to say that I downloaded and synced fulcrum about a month ago and had syncing problems. I tried syncing fulcrum on bitcoin 3 times and it would crash somwhere after block 400,000 after running out of memory. I tried disabling autosuspend as suggested here, but the sync still failed. After disabling fast sync however, the sync was successful. |
|
Thanks for the feedback @jonscoresby ... Perhaps I need to go back and see if I can make |
|
Just wanted to swing by and report that I seem to have the same problem: I tried to index on a Raspberry Pi with 4GBs and it flooded my RAM up till the point that it exited with the above error message (first try with So I disabled fast-sync and now its humming away slowly but steady at 2.7 GBs (baseload was something like 1.3GBs of RAM). |
|
Yeah I really need to set aside some time and have the fast-sync option auto-detect this situation on OS's such as linux that overcommit and prune it down if that happens. I definitely will work on this soon! |
Sorry I didn't see this. I do not have a swap enabled. |
|
Ah I see. Thanks for getting back to me. Yeah I have a hypothesis that this is more likely to happen in the "no swapfile" situation. I am not sure why it became fashionable to ship Linux installs these days with no swap. I remember a time when every Linux install had a default swapfile setup. At some point that changed. Anyway -- I think that in the no-swapfile case, memory usage can get out-of-hand temporarily with --fast-sync and rocksdb both gobbling up RAM. And, of course, if there's no swap.. when you are out of RAM .. something must die. And that thing is Fulcrum. I can't fully control memory usage (because rocksdb lib does its own thing and sometimes overallocates memory temporarily even when you tell it not to). I can, however, mitigate this by detecting the situation and controlling the --fast-sync memory usage .. if it looks like we are reaching the system limit, I can just prune the cache temporarily to be smaller than what the user specified.. or something like that. |
|
Also no swapfile on my linux. Let me know if I should test something once you might have figured out a solution. |
|
Please, please, please fix this. We are trying to package Fulcrum for embassyOS and this makes the otherwise amazing experience very painful. It can take several days to build the index on a low-resource device in docker, and to be told that you have to do it all over again is enough to make the user want to simply delete it and switch back to electrs. |
|
I will fix in in a future release, that's the plan. Please don't use |
That's great to hear. I've also noticed that That said, I do see this issue mentioned more frequently not for the initial sync, but for accidental power loss or other ungraceful shutdown conditions. |
Excellent, great to hear!
This problem does not only present during initial sync. We have already experienced corrupted databases on a couple of devices that were already synced. |
|
Any update on this issue and #155. Start9 is still very excited to get Fulcrum onto StartOS, but not as long as ungraceful shutdowns necessitate resyncs. Is there any update on that issue as well as the issues related to "fast-sync" discussed above? |
|
I lost power this morning and my Fulcrum database is now corrupted. It took several days to sync on my SSD. For the time being I will switch back to electrs until this issue is resolved. |
|
Hello 👋 I ran into this problem when trying to synchronize my Fulcrum Server. The process was consuming too much RAM until it was killed by the OOM Killer (Out of Memory killer), causing the program to be closed forcefully, and corrupting my I tried with different settings in And it always ended up failing and corrupting the db. For context, this is my setup: I compiled Fulcrum myself following the instructions detailed in the project's README.md and I didn't understand why this was happening, as it's not the first time I've synchronized a Fulcrum Server and it's never happened to me before. As on other occasions I used images already compiled from the project and this had never happened, I thought it must be related to the way I compiled the project. It was then that I noticed this:
To solve the problem, I installed I verified the installation by running: Then i verified if the flag for jemaloc exists by running: This should return Then I recompiled the project. To do this, I ran the following commands: This should return somethis like this: Then i run the following command to execute the Then just run: Finally, to check if And you should see something like: Since my Fulcrum installation is using Hope this helps 🙏 |
Great SPV server! It is fast and just what I needed.
I've encountered a problem when restarting Fulcrum:
I restarted the server (via systemd) and also created an image snapshot for the server and then this error appears.
Having used Electrum Cash for a while, I never had any problems with this before (for years) even thought I would restart/terminate instances abruptly.
Is there anyway to prevent this problem from happening or gracefully recover? Thank you!
The text was updated successfully, but these errors were encountered: