Recent data on Turbo-Geth performance
One of the test nodes finally synced to the tip of the mainnet, so I took snapshot of things at the block 5'828'433 (mined June 21st 2018 around midday UTC). The HDD test node I am comparing with is still syncing.
Horizontal blue lines are placed every 24 hours, so you can see it took more than 12 days to sync. Most probably, the slope changes angle every time there is a gas limit increase or spam attack in 2016.
Database and disk usage
In total the database size after syncing to the block 5'828'433 (mined June 21st 2018 around midday UTC) was 341 Gb. And here is the breakdown:
However, this particular test was running with the code that did not yet have receipt storage optimisation. The code that does, is still at the block 5.5m, so I made a pessimistic projection (I took the database size at block 5.5m without optimisation, which was 284Gb and the database size at block 5.5m with optimisation, which was 216Gb, and applied the saving of 67Gb), and this is how it looks:
Having seen how relatively small Accounts bucket is (3.44 Gb), I have tried to rebuild the accounts Patricia tree from it, and it only took 16 mins on a cold machine. The reason it took couple of hours before was that I was rebuilding from the history bucket (current state and history was not separate back then). And this means that I will probably ditch the idea of hash file (remember that 32Mb file, and later I made 512Mb, that contained hashes of the 5th, and later the 6th level of the accounts patricia tree). It will make the code a bit simpler.
Although I was not able to spend a lot of time on Turbo-Geth since my last update, there is a bit of progress on eth tester (and this is the highest priority at the moment). The tester progeam is now capable of connecting to a ethereum node (better run it in “no discovery” mode, so it does not try to find any peers), and “offer” it series of blocks to download. It is sort of equivalent to running “geth import”, but the main purpose is, of course, different. I will describe it as I develop it.