Masternodes: HOWTO solve sync or stuck problems of nodes (VPS / server side)


  • 3DCoinTalk Italia

    This post contains a new useful guide to solve any problems with sync or stuck nodes. These are commands that we have often discussed and used in the official Telegram group.
    Attention, before starting it is important to clarify some essential things.

    Important premises:

    1. This guide is certainly valid for those who have installed the masternode using the official procedure indicated in the official masternode installation guide available both in the Resources - Masternode guide menu of the 3dcoin.io website and through the direct link: https://3dctalk.net/topic/12/masternode-installation
    2. This guide should NOT be confused with the guide for wallet qt sync problems (the computer wallet). There is a separate specific guide to solve sync problems of computer qt core wallet (wallet that does not connect, wallet that remains locked on a certain block, check methods in case of doubt to be on the wrong chain in the rare case of split chain). The guide for problems of sync / stuck of the computer wallet can be found at the following link: https://3dctalk.net/topic/1193/core-qt-3dcoin-wallet-and-blocks-how-to-solve-block-sync-stuck-split-chain-problems
    3. Last but very important premise, something that "node operators" should know well, is that before thinking of having problems to be solved in the masternode that is in the VPS, before this one must be certain that the PC wallet is not giving an incorrect message about the "status" of the masternode. I refer to the well-known fact that to be absolutely certain that the masternode is ENABLED or in NEW START REQUIRED or EXPIRED it is always necessary to delete the "infamous" file "mncache.dat" from the folder containing the files of the wallet in the PC (the path of the folder is visible in the "Tools - Wallet Repair" menu of the wallet, not to be confused with the folder containing the executable 3dcoin-qt). Only after deleting the mncache.dat file and starting the wallet and waiting several minutes and only when your listed masternode passes from the MISSING status to the real status (ENABLED or NEW START REQUIRED or EXPIRED) can you (almost) be certain of real situation of your masternode and then go to the VPS to make the appropriate checks and if necessary apply the instructions of this guide.

    Guide:

    So after carefully reading the three premises above, if you are sure you have sync problems in your masternode, usually connected to VPS via SSH protocol (so with Putty, Bitvise or with the terminal directly, as usual) and proceed as indicated below:

    So you have sync problems of masternode on VPS ? In official and updated installations it should not happen thanks to the new script "daemoncheck" or "antistuck" function introduced at the beginning of 2019, because there is this command linked to a special script that starts automatically every 10 minutes thanks to the crontab and checks and clears caches if necessary:

    /usr/local/bin/Masternode/daemon_check.sh
    

    But if you still have a non-synchronized masternode to unlock manually, alternatively you can try the following commands if your masternode is not in sync or is in stuck

    • Soft method (try this first). Attention please! It's a single string of multiple commands, to copy and paste all at once, all the complete string, in the terminal and then press the ENTER key:
    yes | sudo /etc/init.d/cron stop && sleep 5 && 3dcoin-cli stop && sleep 10 && rm -f ~/.3dcoin/{debug.log,mnpayments.dat,mncache.dat,banlist.dat,peers.dat,netfulfilled.dat,governance.dat,fee_estimates.dat,3dcoind.pid} && sleep 5 && 3dcoind && sleep 10 && 3dcoin-cli mnsync reset
    

    After this first command, if there are no particular reasons for not doing so, I would also recommend rebooting the VPS with the command:

    sudo shutdown -r now
    

    With the system reboot the cron service will also be reactivated and the node will restart automatically.
    Then wait about an hour and double-check if it is in sync with the two classic commands (in sync with the blocks and in sync with the list of masternodes, two different things).

    To check the number of blocks on the VPS:

    3dcoin-cli getblockcount
    

    (and check on 3dcstats.net explorer if the number of blocks corresponds).

    If the blocks should remain too long time at 0 then either restart the VPS as mentioned above or try reindexing with the following command:

    3dcoin-cli stop && sleep 10 && 3dcoind -reindex-chainstate
    

    Or you can use the slower but complete re-indexing method, without -chainstate option, if the first one does not help:

    3dcoin-cli stop && sleep 10 && 3dcoind -reindex
    

    A more precise verification of the sync of blocks can be done with the following command without the need to open the browser but directly only on the VPS:

    3dcoin-cli getbestblockhash && curl -s https://3dcstats.net/insight-api/status?q=getBestBlockHash
    

    the result will be such a thing for example:

    000000000011277c605ff2f7580f80701e7660a378657f957b8166e1499456d9
    {"bestblockhash":"000000000011277c605ff2f7580f80701e7660a378657f957b8166e1499456d9"}
    

    (this result refers to the historical block of the fork 475000 and it's only an example)

    The first and second series of numbers must be the same, if they are not the same try again a couple of times ... if they continue to be different then there is a problem of sync ... in the case in which just at that moment the explorer should be offline, the command may not work, but you can also check the number of current blocks in the information menu of the wallet in your pc. The commands getblockcount and getbestblockhash also work in the "debug console" of pc wallet.

    To check if your masternode are in sync with masternodes list:

    3dcoin-cli mnsync status
    

    (this command must result in asset id 999 and sync finished)

    After these checks, if you are sure you have everything in sync on the VPS (and only in case of need!) you can give a new START ALIAS from the pc wallet after selecting the right masternode line (first as always delete the file mncache.dat from the wallet folder and wait complete wallet synchronization for a quarter of an hour or more after re-opening the wallet).
    ⚠ Warning: Obviously with a new START of the masternode from the wallet you lose the turn of the rewards and you have to get back in the queue. So always consider carefully if restarting from the control wallet is really necessary! ⚠

    • Hard / Strong method (try only in case of failure of the first method). If with the first method or "soft" method (soft because it does not delete all the blockchain) you have not solved your problems then use this additional method below. Attention please! It's a single string of multiple commands, to copy and paste all at once, all the complete string, in the terminal and then press the ENTER key:
    yes | sudo /etc/init.d/cron stop && sleep 5 && pkill -9 -f 3dcoind && sleep 5 && rm -rf ~/.3dcoin/{blocks,chainstate,debug.log,mnpayments.dat,mncache.dat,banlist.dat,peers.dat,netfulfilled.dat,governance.dat,fee_estimates.dat,3dcoind.pid} && sleep 5 && 3dcoind
    

    This method can be defined as "hard / strong" because it "kills" the "3dcoind daemon" and deletes the entire blockchain from your VPS. The resulting sync operation will therefore be slower as the whole blocks database will be re-downloaded.

    As already indicated above, now you need to do all the different steps (already mentioned above after the "soft method" but I repeat now so you don't have to go up again in the post).

    After this first command, if there are no particular reasons for not doing so, I would also recommend rebooting the VPS with the command:

    sudo shutdown -r now
    

    With the system reboot the cron service will also be reactivated and the node will restart automatically.
    Then wait about an hour and double-check if it is in sync with the two classic commands (in sync with the blocks and in sync with the list of masternodes, two different things).

    To check the number of blocks on the VPS:

    3dcoin-cli getblockcount
    

    (and check on 3dcstats.net explorer if the number of blocks corresponds).

    If the blocks should remain too long time at 0 then either restart the VPS as mentioned above or try reindexing with the complete re-indexing method:

    3dcoin-cli stop && sleep 10 && 3dcoind -reindex
    

    A more precise verification of the sync of blocks can be done with the following command without the need to open the browser but directly only on the VPS:

    3dcoin-cli getbestblockhash && curl -s https://3dcstats.net/insight-api/status?q=getBestBlockHash
    

    the result will be such a thing for example:

    000000000011277c605ff2f7580f80701e7660a378657f957b8166e1499456d9
    {"bestblockhash":"000000000011277c605ff2f7580f80701e7660a378657f957b8166e1499456d9"}
    

    (this result refers to the historical block of the fork 475000 and it's only an example)

    The first and second series of numbers must be the same, if they are not the same try again a couple of times ... if they continue to be different then there is a problem of sync ... in the case in which just at that moment the explorer should be offline, the command may not work, but you can also check the number of current blocks in the information menu of the wallet in your pc. The commands getblockcount and getbestblockhash also work in the "debug console" of pc wallet.

    To check if your masternode are in sync with masternodes list:

    3dcoin-cli mnsync status
    

    (this command must result in asset id 999 and sync finished)

    After these checks, if you are sure you have everything in sync on the VPS (and only in case of need!) you can give a new START ALIAS from the pc wallet after selecting the right masternode line (first as always delete the file mncache.dat from the wallet folder and wait complete wallet synchronization for a quarter of an hour or more after re-opening the wallet).
    ⚠ Warning: Obviously with a new START of the masternode from the wallet you lose the turn of the rewards and you have to get back in the queue. So always consider carefully if restarting from the control wallet is really necessary! ⚠

    Conclusion:

    If you do not solve this either, then you have problems in the VPS that are beyond the 3DCoin masternode, first solve the problems of your VPS and then try again, I remind you that it is necessary to use KVM or VMware type VPS ie with full virtualization (avoid OpenVZ / OVZ types VPS because it is not a system suitable for manage a masternode for a normal not skilled user). I'm not talking about choosing a provider, but choosing the type of VPS, they are two different concepts.

    If your VPS is of the "right" type (and maybe your node has also worked for some time and you have also perceived rewards before) and you cannot now solve the problems by following this guide then probably the best choice and "faster" or painless is to "reinitialize" the VPS from the provider panel (thus "reset" it completely using the options provided by the provider) and then proceed with a new installation following the official guide (it is not necessary to repeat the collateral send or create a new masternode private key with the "masternode genkey" function of the wallet, you can reuse the same, you can see masternode key in the masternode.conf of your pc). In case of doubt, if you want to reinstall your masternode you can consult the following guide:
    https://3dctalk.net/topic/1155/masternode-problems-howto-uninstall-and-make-a-clean-installation-the-classic-red-panic-button-when-you-do-not-have-other-solutions

    Useful official masternode installation guide:
    https://3dctalk.net/topic/12/masternode-installation

    For more information or if you need assistance do not hesitate to contact support here in the 3dctalk.net forum or in the official Telegram group
    https://t.me/Project_Districts

    Enjoy and have fun with 3DCoin masternodes! 😉