Release: PolySwarm Gamma (v0.3)

PolySwarm Hive (testnet), community-developed microengines, searchable worker registry, and assertion confidentiality.

The PolySwarm team is thrilled to announce the immediate availability of PolySwarm Gamma — the last major release before Stable (v1.0)!

This release we’ve been focused on PolySwarm’s market participants (ambassadors and microengine developers) — so beyond delivering on the road map, we’re excited to share some of the progress we’ve made:

1. We’re in various stages of integrating a dozen engines into the Hive testnet! These are engines from small to midsize AV companies as well as three independent researchers. This is exciting! We’ll be making joint announcements introducing these initial participants in the coming weeks.
2. The PolySwarm grant program is humming along. We’ve received (and funded 3 so far) some fantastic ideas! If you have a great idea for detecting malware, we want to hear it. If we agree it’s a cool idea, we will give you a minimum of $5000 USD to build it. You retain copyright and intellectual property — you just have to run the microengine on the PolySwarm network. Apply here.
3. We are constantly improving developer documentation and making it easier for micro-engine developers to focus on detecting malware.

Let’s get into the Gamma details, but first a quick word about the “hacker summer camp”!

PolySwarm in Vegas for BlackHat USA & DEF CON

Go to Las Vegas in August, they said. It’ll be fun they said. Well, they were right.

We’ll be at BlackHat and DEF CON this year (Aug 7–12)!

If you have a novel idea for reinventing malware detection — perhaps analyzing JavaScript in PDFs or macros in Microsoft Word or native libraries in Android apps, we want to meet you — and potentially fund your idea. We’re in Vegas to engage with security experts like you at the two largest InfoSec conferences of the year!

Visit us the PolySwarm booth (booth #CZ209) at the BlackHat career zone next to NVIDIA, Cisco and Amazon and chat us up online or in-booth for an invite to the catered (food & drinks) PolySwarm party on August 10th.

We’re handing out literal blockchains at the PolySwarm booth. That’s a real, tiny cinder block. You want this.

And don’t forget to pick up your (literal) blockchain!

Gamma: The Nitty-Gritty

Enough about Vegas.

Gamma brings a host of new features, bolsters & extends existing features and — most importantly — lowers the barrier to entry even further for microengine developers and ambassadors.

PolySwarm Worker Registry

PolySwarm Gamma offers a worker registry.

A new handy CLI makes it super easy for developers to generate a Registry JSON and advertise their microengines.

The Registry is a sort of “classifieds section” that allows microengine developers to advertise their wares by publishing a JSON file that describes their microengine(s). Ambassadors can consult the registry in order to discover expertise that is relevant to their current needs.

The Registry acts like Google Play in the sense that a microengine is tied to its owner through cryptographic signatures:

• Each microengine and each developer ID has a corresponding cryptographic keypair.
• Developers claim ownership over microengines by adding claimed microengines’ public Ethereum address(es) to an ownership array in the JSON.
• Ownership is proven by signing the corresponding item in the ownership array with the private key of the microengine itself.
• This JSON is hosted on IPFS.

Seemingly a simple feature, the Registry is used as the basis for attributing microengine performance and reputation to deserving developers. If the developer so chooses, we also plan on allowing microengines to be tied to a real world identity in PolySwarm, enhancing enterprise trust.

Plenty of behind-the-scenes changes across the board since Beta.

Pre-Deadline Assertion Confidentiality

Confidentiality of assertions is a concern raised by many. In a nutshell: we must prevent microengine copycats from duplicating the assertions of their hard-working competitors, effectively diluting the NCT rewards of value creators.

The solution we came up with is a cryptographic commit / reveal scheme. Here’s how it works:

1. A bounty is placed and its assertion window opened.
2. A microengine decides it would like to assert on the bounty.
3. The microengine chooses a random number (a nonce) and computes: hash(assertion_bitfield + nonce + msg.sender) = commitment, where hash is a cryptographic hash function.
4. The microengine commits to their assertion by writing commitment to the chain.
5. The bounty's assertion window closes — no further assertions are accepted. Simultaneously, the reveal window opens.
6. The microengine reveals its assertion by publishing assertion_bitfield and nonce within the reveal window. If it fails to do so, the microengine is assumed to be wrong and loses its staked NCT.
7. With the revealed assertion_bitfield and nonce, all participants can verify that hash(assertion_bitfield + nonce + msg.sender) = commitment.

This scheme accomplishes two things:

1. It proves the microengine committed to assertion_bitfield (due to second pre-image resistance of the hash function) during the assertion window — locking in their assertion.
2. Absent a leak, competitors are unable to observe the assertion_bitfield during the assertion window, and therefore can not exploit it to copy the hard working microengine.

Arbiter Selection & Ground Truth Updates

We’ve further improved arbiter selection & ground truth determination algorithms. This deserves a blog post & visuals of its own. Highlights:

1. Arbiters must stake a large amount of NCT in order to remain eligable.
2. We’re working on a DPoS (delegated proof of stake) model for allowing the community to choose arbiters.
3. A similar commit / reveal scheme for ground truth votes that prevent arbiters from piggybacking.
4. A means to ensure minimal ground truth determination time without economically encouraging arbiters to shortcut their analyses.

PolySwarm on all the Platforms

We want PolySwarm to be as simple as possible for external contributors, microengine developers and ambassadors to use.

Toward that end, we’ve started publishing ready-to-go PolySwarm resources on your favorite developer outlets:

1. We’ve Dockerized (almost) everything — and we’re auto-pushing the latest passing builds to Docker Hub. This makes things dead simple. Want to run polyswarmd? Just docker run polyswarm/polyswarmd. Done. Want to run an entire PolySwarm ecosystem on your own machine? There’s a one-liner docker-compose for that. No cloning repositories and no building anything. You focus on writing your microengine, we’ll handle the rest.
2. The worker-registry CLI is available directly on npm. Grab it and go.
3. And of course, all PolySwarm source code is available on GitHub. In the next few weeks, we’ll be switching primary development to GitHub, allowing us to more easily accept external pull requests and collaborate with others on issues and tickets!

Did we miss something that would make it easier for you, the developer? Get in touch via Discord and let us know.

End-to-End (e2e) Testing

Necessary, but not particularly exciting, merges into master across almost all PolySwarm projects must now pass end-to-end automated testing.

This allows us to confidently ship source and Docker images that are tested to play well with one another.

A multi-stage test conducted on the worker-registry repository.

This was particularly important due to the large number (13 as of this writing!) of components that represent the PolySwarm Gamma release.

An end-to-end test of the contracts repository.

Project Consolidation & Simplification

Building an entire marketplace requires lots of moving parts, which manifests in many separate projects.

We know this can be confusing at times, so we’ve consolidated some projects to make it easier for newcomers to wrap their heads around the PolySwarm ecosystem:

1. microengine: Various microengine projects, such as microengine-clamav have been merged into a single microengine project. This project will house all of the community-maintained, F/OSS (free/open source software) microengines. Rather than having disjoint projects, we now have fancy sub-classes for each backend.
2. contracts: We experimented with separate contracts and contractor projects. The latter was responsible for deploying the former. This turned out to be unnecessary segmentation; we’ve consolidated the ability to deploy the contracts into the contract project itself.
3. transaction-signer: A stopgap project during a refactor of the polyswarmd API. These changes have been integrated into arbiter, ambassador and microengine.

Onward to Stable!

Follow us on Twitter, Telegram and GitHub.

And, we’re sharing a special Gamma invite to Discord for the first 50 people who click through with this link!

See you in Vegas!