govBuy: learnings from round 1

With some sweat and tears, we’ve finally concluded round one of the govBuy experiment. Here’s a summary of what we’ve learnt.

Overview of the experiment

We forked 18F’s repo and stripped away stuff which was not applicable to us, such as the SAM.gov integration (it’s a vendor registration system in the US, and you had to be registered before you can take part in their auctions). This was the cheapest way for us to test its feasibility in Singapore.

We chose to put up the simpler and lower priority tasks in our backlog, to allow more people to participate, and to minimise the risk to our project. Due to the simplicity of the tasks, we were able to keep starting bids of the tasks low, allowing us to use the simplest procurement process, as well as minimising the financial risk.

Publicity

We started with an announcement at NUS Friday Hacks, and followed up with mailers to all the local universities, as well as announcements to some of the local developer meetups, such as the Ruby meetup and React meetup. Broadcasting to the local developer meetups was the most effective, so we will continue to tap on that channel. Now that we have a user base, we will also email them of new opportunities on govBuy.

Bidding

18F’s platform supports two auction types: sealed auctions and reverse auctions. The main difference between the two is sealed bid prices are only revealed after the bidding period ends. Furthermore, reverse auctions allow users to place multiple bids. Tasks for PR Demon and License Finder used reverse auctions, while tasks for RCanary and ESLint used sealed auctions.

An interesting observation was that the reverse auctions ended up at about ⅓ of the starting price, while the sealed auctions ended up at ¼. This was somewhat counter-intuitive, as many people expected a “race to the bottom” for the reverse auctions, which actually didn’t happen.

The bidding time for both auction types were similar, with most of the bids placed at the start and end of the bidding period. Therefore the bidding period can be much shorter. Some of the feedback we got from users was that the long bidding period deterred them from bidding, as it was hard for them to plan their schedule, not knowing if they would win the auction.

Profile of Bidders and Winners

While we had a good mix of bidders from both the industry and students, the winners were mostly students. While the small number of data points we have is not conclusive, this is certainly an interesting statistic to keep an eye on.

Delivery Timeline

To encourage more bidders, we deliberately set a long delivery timeline of 2 weeks, even though we knew the actual work could be done in a day. We were happy that all the tasks were completed on time, with most of them done in the first few days. As such, we’ll be using more realistic (read: shorter) timelines in the future.

One of the tasks was only submitted in the last few hours though, and that caused us a bit of panic as the solution wasn’t ideal. Luckily the bidder was able to incorporate our suggestions quickly and managed to deliver the task on time.

We hope to encourage early submissions (even incomplete ones), so that we can work together with the bidders to deliver a workable solution for the task.

Code Quality

The submissions generally fulfilled the functional requirements but there was a bit of back and forth to help the submitters clean up their code. We received feedback that some of the requirements were not clear but the clarifications and code reviews on GitHub compensated for that.

We also realised that the non-functional requirements could be less subjective (e.g. ask for 99% test coverage instead of just asking for “appropriate tests” to be written).

In some of the submitted pull requests, instead of explicitly pointing out code smells in the GitHub pull request discussion (such as unhandled conditions or exceptions), we found that implementing linters could be a more straightforward and helpful way for the submitters to understand our code quality and style standards.

Payment

Our payment method evolved over the course of the experiment. While we initially thought of issuing cheques to the winners, new instructions from the Ministry of Finance recommended GIRO payments instead. So we got the winners to fill in and snail mail the GIRO form to us, along with proof that they owned the specified bank accounts. This seemed like a lot of hassle for the winners, so we tried to find a simpler way to pay them.

In the end, we paid the winners from our personal internet banking accounts, and they only had to email us a signed receipt. While we were able to get payment to the winners sooner, we had to manually submit a staff claim. This is clearly not sustainable in the long run so we’ll continue to simplify this further.

Conclusion

With such a small sample size, it is too early to say that we’ve validated our assumptions that auctions for open source work is viable in Singapore, and that there is sufficient interest from the developer community. It’s also not very conclusive whether the sealed or reverse auctions work better, so we will continue to try both auction types. However, we have opened a way for individuals to help the government deliver digital services, which wasn’t possible previously.

Based on our observations and feedback from users, we’re definitely going to shorten the bidding and delivery timelines to a week or less for future auctions. We’re also thinking of trying out bigger tasks and non-code work (e.g. design) on the govBuy platform. It will be interesting to see how it’ll work out (or not).

Beyond govBuy, we also hope to inspire the rest of the government to experiment quickly and continue to improve the way our government works. Leave some comments below or email me at chai_sheng_hau@tech.gov.sg if you have any ideas to share.