City of Zion Project Report — June to August 2018

Published in

Proof of Working

12 min readOct 6, 2018

City of Zion (CoZ) is a global community of open source enthusiasts, with the shared goal of helping NEO achieve its full potential. CoZ primarily operates through the community Discord and CoZ Github, central places where the community shares knowledge and contributes to projects.

CoZ is neither a corporation, nor a consulting firm or a devshop / for-hire group.

We apologise for the delay in posting the latest project progress update, we have been busy! Finally, here is the report for the past few months written by the project maintainers themselves.

neo-go

https://github.com/CityOfZion/neo-go

neo-go rewrite to Version 2.0 started

After having a much better understanding of blockchain technology and how NEO works, it was time to start over the core parts from scratch and implement it with the correct semantics and code engineering practices.

Created package wire which will handle the network protocol
Created new crypto package
Started writing the core fundamentals for the light node
Made each module, independently importable

Start of project neo-storm

Link to the Github repository of neo storm

After NEO announced that it’s going to change some core semantics on the NEO protocol, I was thinking that developers would be better off with a nice and easy to use framework for smart contracts. It makes more sense to focus on smart contract development than continuing writing the port to Golang. Currently almost all NEO developers are using Python or C# for writing their contracts. However, some of NEO’s strongest points are the ability to write smart contracts in multiple languages, so why not improve the Golang compiler so it can be equal to it’s brothers (Python and C#).

neo-storm is more then just a Golang to AVM compiler

Golang to AVM bytecode compiler
Fully stubbed NEO virtual machine (no node needed)
Smart contract debugger
Tooling (SDK) for deploying to test and production environments
Package manager for smart contract modules that are written in Go

Finished neo-storm version 1.0

Neo storm reached it’s release candidate 4 which makes it a fully working Golang to AVM compiler. This is not yet suitable for production due to not being fully stress tested.

Start of the new improved VM

Started a new version of the VM with improved security and the ability to work fully independently. This means that developers using neo-storm will not have to spin up a private net to be able to test their contracts. The VM will have all runtime and NEO API’s built in and mocked, like it was a fully working blockchain.

neo-sharp

https://github.com/CityOfZion/neo-sharp

In June, the neosharp team, consisting of 8 members, continued to build out the platform particularly around cryptography, network messaging, and persistence.

A major milestone was met as the team achieved generation of the Genesis block and began testing the persistence of blocks from both testnet and mainnet sources. There were also continued efforts around network messaging, unit test coverage, and updates to contribution guidelines and best practices to enable new contributors to participate in the project and get the codebase ready for the sprint in Shanghai to work alongside NGD resources.

In July, the rubber really started to hit the road with neosharp and the Shanghai sprint. The team gained 5 new contributors and had almost 100 pull request / commits in the month of July.

Major areas of focus in this month were integration of wallet features, application and console features, addition of the task scheduler framework, RPC implementation, VM integration, block processing / persistence, and unit test coverage. The framework was also upgraded to dotnet core 2.1 and the Shanghai sprint was a success as the beginning of NGD resource integration into the neosharp development efforts, with expectations of deeper integration and resource allocation to the project in the coming months.

In August, the contributor base grew further to a total of 13 regular contributors / participants in the design, testing and implementation of neo-sharp. The project management was moved to use Github issue tracking and project management, sunsetting the legacy Trello board used by early contributors.

There was also an increased focus on test coverage, with a new initiative started to leverage containers to facilitate functional testing of nodes in a network configuration. The use of dynamically allocated containers will enable developers to the ability to test nodes in a way that has yet to exist in the NEO community to date.

In addition to testing, persistence was further implemented, bringing the Redis DB persistence module functionally up to part with the RocksDB implementation, and the first JSON persistence module was implemented using Redis to allow for more convenient third party interop and debugging for developers.

neoscan

https://github.com/CityOfZion/neo-scan

Switch from exoscale hosting (5 servers for neoscan.io, neoscan-testnet.io, coz.neoscan-testnet.io, nex.neoscan-testnet.io) to a more unified deployment on Google Kubernetes Engine cluster enabling faster deployments / rollback / database backup / replication / tests.
Improve neoscan synchronization time (from 4 days to 24h) using blocks parallel insertion.
Improve unit testing stability and speed
Various bugfix/refactor
Add a deprecated tokens list to display it on the UI (e.g. NRVE used in favour of NRV token)
Update neoscan documentation and generate it automatically from unit tests
configure and create deployment scripts for all environment (neoscan.io, neoscan-testnet.io, coz.neoscan-testnet.io, nex.neoscan-testnet.io) so all those endpoint receive latest update from neoscan)
faster block consistency check at startup
use a segment_tree structure for address gas calculation
various bugfixes
The neoscan code base has been entirely rewritten with a new postgresql data structure giving the following improvements:
faster blockchain syncing (2 times faster)
better data consistency / crash recovery scenario
exhaustive tests (more than 90% code coverage)
better maintainability (remove code redundancy, codebase reduced of 25% LoC)

Neon Wallet

https://github.com/CityOfZion/neon-wallet/releases

In June the foundation and design of the new UI/UX for v2 was created and carefully reviewed. Under the hood performance and security updates were made to both v2 and the existing versions of neon-wallet. A project board was created to enscapulate the items representing v2 MVP. Network fees (priority fees) were introduced into neon wallet.

Implementation tx network fee (priority tx) in neon wallet v1
Many iterations of product design review on new v2 concept
Creation of the v2 project board on gh and managment of the stories
Electron security fixes in v1 and v2 branch

In July the common component library for v2 was developed for use throughout the application and many internal APIs were simplified and refactored. Work was done to no longer necessitate a release for every new token that gets added to the network and a few new contributors were onboarded and contributed great work. Authentication, send and wallet manager UX was implemented and QA’d in neon wallet v2. While development on v2 was being done there was also a patch released for the existing version of neon-wallet in production (0.2.6).

Introduction of font family from design spec to application
Implmentation of all authentication UX/UI related to the new design
Audited and fixed type annotations
Styled and created general common components for use by future contributors
Introduced logic into v1 and v2 for fetching token hashes dynamically via neo-tokens repo (previously a release was required to add hard coded script hashes)
Work on voting UI and redux implementation
Onboarding new contributors
Audit on the release process and circleCI pipeline creation of release documentation https://gist.github.com/comountainclimber/a0dd9f7553ffd75ad7a3635c7ea70d5b (previously non existent)
Fix of broken HMR webpack implementation
Release of 0.2.6 https://github.com/CityOfZion/neon-wallet/releases/tag/0.2.6
Update of core dependencies
v2 “send” UI/UX
implementation of v2 “wallet manager” UI/UX
iteration and review on design mocks related to “empty” wallet state

During august the team focused on more UX/UI design implementations in v2. The v2 dashboard components were fully implemented including token balances with token logos. Work was completed on the Settings, node selection flow, Token sale, receive, nep9 QR generation functionality, transaction history and contacts sections of the new application. Many of the existing UX flows were further polished. Assets including icons and images were updated in implemented throughout the application. There were numerous under the hood updates to the state management layer of the application. In addition to all of this hard work, during this time two releases were cut for v1 0.2.7 and 0.2.8 these releases introduced numerous performance enhancements the biggest one being a dramatic reduction in newtwork calls needed to support neon-wallet. These releases were also ported back over to v2.

Release of 0.2.7 https://github.com/CityOfZion/neon-wallet/releases/tag/0.2.7
research on implementing network fees in token sale UX flow
Polishes v2 “send” UI
Bug fixes
Fixed/simplified dynamic hash logic and updates hard coded token list
Fixed broken links in transaction history when using coznet
Port of release 0.2.7 into v2 branch
Leverages the dynamic token hash work to include images for nep5 tokens on dashboard
Audit of v2 branch at varying screen resolutions
Creation and polish of dashboard components (v2)
Coordination with design on nep9 mocks (v2)
App icons (v2)
Adds the ability to minimize the application via keyboard shortcut (v2)
Adds version num dynamically from package.json in auth view (v2)
Release 0.2.8 https://github.com/CityOfZion/neon-wallet/releases/tag/0.2.8
Complete rewrite of the dashboardActions getBalances logic. Leverages the new getTokenBalances functionality in neon-js to dramatically improve performance and simplify the code substantially
discovery of and fix of invalid script hash for SOUL in neo-tokens repo
Many hours spent on general code quality improvements — fixing linting and type errors introduced by previous contributors / maintainers
Fixing bugs in the CI pipeline
Implementation of priority fees in v2 send

neo-python

https://github.com/CityOfZion/neo-python

In June we polished several parts related to Transactions. The parsing was updated, we added the ability to add transaction attributes during testing and actual invocation and a new command `tkn_history` was added to show all historical transfers of the specified token. Just to name some changes.
On the utility side we added `np-sign` for…well signing stuff using a wallet, NEP2 formatted key or WIF. Furthermore `np-export` and `np-import` have been introduced for raw blockdata backup and restoration. The RPC server received some love by allowing it to run in parallel with the REST server, fixed some bugs and adding a couple of new RPC methods to get it closer to feature parity with the Official C# RPC Server. The last cool feature we’ll highlight are the `Iterator` APIs for smart contract development that allow you to do prefixed searching in the storage context. Have a look at the changelog for the full list of updates!

While July is generally a slower month we did not sit still. We’ll highlight a couple of cool and important updates. First off the command prompt now supports the ` — i` flag that allows you to interactively specify the arguments and their types during invoke calls which greatly improves parsing and control of what goes into your invocations transactions. NEP5 Token sending from multisignature addreses was not working which means you could not transfer your NEO Testnet funds using `neo-python`, fixed! A couple of network related improvements were made that should give better availability of connected nodes. The remaining updates have been mostly internal work to improve stability and correctness of data structures when compared to the C# client. As always, have a look at the changelog for the full list of updates!

August, the end of the summer is nearing. We did some more updates on the networking side now for node selection. The ability to add a fee to transactions was added such that you can prioritize your transaction processing on the network if you’re willing to pay for it. We added support for newer library versions to stay up to date with the improvements, expanded our available RPC methods, synced up with the official node GAS calculation updates and squashed a couple more bugs. Most notable of the bugs is we fixed and issue with header importing of the `np-import` tool. For the full list of changes have a look at the change log!

neon-js

https://github.com/CityOfZion/neon-js

In June, the first beta version of the new version 4 was released. The new chosen stack includes typescript, lerna, jest and yarn. This would give us the tools to manage a growing codebase of multiple packages. While the aim is to have multiple packages, the neon-js package will still remain as a functional library with the same functionality.

The core package will be released under the name @cityofzion/neon-core instead. This will feature only the named modules and none of the semantic API. The api functionality in the current package will be extracted out as @cityofzion/neon-api and @cityofzion/neon-nep5 plugins that augment the core package. (Currently the plugin packages are not released and thus not available for trial).

To install version 4, you will need to install it using the next tag:

npm i @cityofzion/neon-js@next

For version 3, we will still continue to release while version 4 is built. This month, we released 3.9.1 which introduced deserialization of complex objects from the VM.

Another release of v4 was made, upgrading the dependencies and focusing on fixing the bugs that occurred during the transition to typescript. On v3, we released 3.9.2 which fixes various bugs that was affecting how neon-js was determining good rpc nodes.

With the announcement of voting, some focus was placed on investigation and putting in support for State Transactions. These are transactions that are used to vote for candidates. This work will be released when the feature is more stable.

Work on version 4 is wrapping up and will soon be released (early September). Work on v3 will stop once version 4 is out and a branch archive/v3 will be carved out to support future patches for v3. Due to the large changes happening in version 4, I do not expect an immediate uptake on version 4. Thus, version 3 will continue to receive patches and appropriate upgrades when necessary. However, version 3 might not have the infrastructure to support the new features that can be published in version 4. For example, any node based packages cannot be ported to version 3 without breaking browser compatbility.

Meanwhile, v3.11.0 is out with several new features. getTokenBalances contributed by David which bunches multiple token balance calls together, allowing for lower overheads. State Transaction support is also merged with setupVote being the managed method for easily setting up your votes. Lastly, verifySignature has been added to complement the signing methods.

neo-local

https://github.com/CityOfZion/neo-local

The last few months have seen the number of users of neo-local (NL) increase due to popularity and success of nOS. The nOS core team released nos-local which imports and uses NL as it’s foundation. As part of the new users, a number of bugs have been highlighted and fixed, and further features have been suggested.

A project which has been extensively discussed on Discord has now been released and added to NL: neo-local-faucet. This new service makes up the 6th container to be added to the NL Docker Compose stack. It is a simple Javascript web application that allows for the creation of a new wallet and the automatic request of NEO/GAS to an account.

Further effort has been put into the reliability of NL, with strengthened CI tests and nightly monitoring.

August saw the start of work towards a large milestone for neo-local, a cross platform CLI! The reasons for switching to a CLI are:

Remove the need for a Makefile as it does not run on Windows (see issue #8 raised by @canesin).
Allow for the project to be easily extended so that more functionality can continue to be added.
Written in Go for better performance and the ability to be unit tested.
Remove the dependency on Docker Compose.

This is a complete rewrite of the project so will require some time. An initial release is aimed for October, and is being worked on by @revett and @jeroenptrs.

A small fix from @jeroenptrs improved how CI tests were run on the project, to improve reporting. Discussions within the neo-local contributors continued on the topic of pushing the project forward, see this gist written by

Jeroen Peeters

Happy hacking!
The City of Zion Community

Follow CoZ on Facebook, Medium Publication and Twitter for the latest updates on CoZ development!

Governance

There is no formal process in joining CoZ. Interested individuals will have to do the work first, and the submitted code has to be licensed under the MIT License. Consistent contributors will become eligible to join the CoZ organization, and begin collaborating on projects. The governance process of this organization can be found here.

CoZ is not a regulatory body and take no responsibility for the quality of 3rd party smart contracts deployed on the Neo Blockchain.

Funding

Since its inception, the CoZ Council has been in close contact with the NEO Council. The NEO Council provides support to the community, in the form of NEO rewards, which CoZ redistributes weekly.

For individuals who are interested in financially supporting CoZ development, our donation address is AXSoNQEKjmqPBNPg5cNrHyWivfjok3Vj9D.

CoZ Council members and maintainers are excluded from weekly NEO rewards, and Council members are excluded from any CoZ competitions.