Last week during a period of maintenance and while support was doing a rescan for missing KLKS transactions an outage occurred. This article is to give visibility into what happened, the actual damage and what we did to prevent this particular issue going forward.
Cryptobridge identified strange transactions in the KLKS wallet being credited. During scanning the KLKS chain for missing KLKS transactions, an agent modified the rescan script to check as far back as 4 months instead of the last 4 –7 days. During this time the BTS node that was being used to check if KLKS had already been issued also had an outage inadvertently. As a result old transactions that occurred during the KLKS swap got added as new transactions since the old transactions would not have been cached.
Upon further investigation it was discovered that the KLKS wallet does not read all parameters in the config file like most other coins. It ignored staking=0 in the config file which caused KLKS to generate staking transactions momentarily. This was what prompted us to start manually scanning the wallet to check transactions which lead to the issue.
What We Did
Withdrawals were halted immediately upon discovering the error that occurred. CryptoBridge then proceeded to do the following:
- Temporarily stopped withdrawals of KLKS.
- Investigated and removed old transactions.
- Recovered the issued KLKS before most could be sold or withdrawn.
- Investigate the root cause and made the changes required.
We’ve recovered all but 115,611 KLKS, worth ~1.27 BTC ($8,255 USD) all of which has been paid back using our coin listing fee.
Our Solution To Prevent This Particular Case From Reoccurring
- Updated the blockchain rescan script to only scan for 7 day periods when required which will always be in the cache and not require a BTS node.
- Developed and open sourced a Bitshares health check module that will be integrated into our processes.
The module can be found at:
In summary it was a human error combined with a temporary outage with a Bitshares node. Despite being in beta we have no excuses for what occurred and have been taking several steps to improve the stability of our infrastructure and lay the foundation for federation (our distributed gateways run by BCO holders). Thank you for your understanding and support.