Updates on Hacker Experience 2
Before you start…
This post got way longer than I expected. But please, take the time to read it in its entirety, for it contains important information about the game, refunds and what to expect in the near future.
I started writing this post in June and finished on September (apparently I’m slow at everything I do), but I’m only being able to publish it now, in December. During this time (September to December), a lot has happened and I’ve appended an update at the end of the post. Because of that, some dates may be wrong. I decided to keep the dates unchanged, along with the original writing.
Dear backer,
It’s been so long since our last update that many of you are wondering whether the project is still alive, whether we’ve gave up or what is going on. This is a rather long blog post where I’ll go over many things that happened to us ever since we were partially funded, two years ago.
First things first: while A LOT has happened, we are still working on Hacker Experience 2, as we’ve been since January/2015. The teams may have changed (except for me), but for the past 40 months we’ve always been moving towards a dream: an open-source hacking simulation game, with realistic and highly strategic gameplay, real-time information flow and location-based features.
I remember our first year (and first team) of development. We’ve created a very fun prototype using Python (and the then revolutionary asyncio module) and React as a front-end technology for our virtual desktop. After ~12 months and ~50 thousand lines of code, you know what we did? Threw everything away. Not because the code was bad, or the game design was flawed. We did so because we realized, based on that year’s experience, that we could change some fundamental mechanics and create a vastly better game.
At the time, I was faced with an important decision. Should I throw away one year’s worth of work? Start from scratch?
We (me and you, backer) have a clear objective. We want a simulation game that goes way beyond the basics.
We want a game that fits well with all player’s profiles. The player will be able to dig so deep into the game mechanics that he or she will learn computer science concepts while doing so. The player will have a clan with corporation-like hierarchy that may be used to share strategies and conquer territories together.
We want a highly strategic game. One that makes the player think on the upsides and downsides of every move, for each action has its consequences. A game that gives the player more than one choice to obtain the same result, each with their own pros and cons.
We want a game with an exciting and immersive story. With compelling interactions between the player and the environment. An interactive story filled with unexpected twists, funny moments and a breathtaking narrative.
We want a game whose code may be freely used by others to build completely different games, without having to re-implement existing features. A game that is open, and that can be extended by its own players.
We want a game that is free, as in freedom. Freedom to run the program as you wish, for any purpose. Freedom to study how the program works, and change it as you wish. Freedom to improve and redistribute the software¹.
We want a game that, once purchased, you actually own and have full access to. A game without pay-to-win or shady mechanics. A game that you can host yourself and play with your friends.
And — for the *NIX geek out there — a game that you can play on your terminal, over SSH, or using Vim-like keybinds :) (a project aptly code-named “HEmacs”).
I just knew, back then, that that game wouldn’t meet these objectives. It could be OK, it could sell enough to make a return on my investment, but it just wouldn’t be awesome.
I don’t regret throwing it away. And no, we didn’t start from scratch. We had a 1-year experience which was essential to evolving our game design. Without that year’s work, we wouldn’t be able to design the mechanics we currently have. This was probably around January/2016.
One year later, on January/2017, guess what? We threw everything away again :) This was another crucial iteration on our development process. It brought some improvements to our game mechanics, and it allowed us to completely change the technical side of the game. We switched languages, technologies and infrastructure. This technical improvement, while transparent to the user, enabled us to design a code that is highly maintainable, testable and easy to expand.
For the past 18 months we’ve been working on this last iteration — and mind you, this is the last one! There is no need to throw this one away, as I feel we’ve reached a maturity on both the technical and game design sides.
Looking back, I wonder. If I hadn’t started over twice, maybe the game would have already been released. You’d have played it already. I wouldn’t be feeling guilty for being over 2 years late on my promised release date. I wouldn’t have spent so much money with the development process, and possibly I could even had earned some money over it.
But no, I don’t regret it. I stand by my decisions, as I believe we can build a truly great game — the game we want!
Why there were no announcements or updates about the game in the past several months?
It doesn’t matter why. It’s completely inexcusable for a crowdsourced project to go so long without an announcement or progress update. It gets even worse when this lack of updates has been happening, on and off, since the project started.
Throughout this post I’ll try to paint the most genuine context of how things have progressed, and maybe this explains the why, but I don’t think it justifies the several months I went dark. As I said, I think it’s unjustifiable, regardless of what happened in the meantime.
I know I must have upset, disappointing or frustrated many of you with this disturbing silence. And for that there’s nothing I can offer other than my most sincere apologies.
These apologies are void if they do not come with a plan to remedy the situation. From now on, many things have changed. Albeit slightly technical, I started publishing a Changeset of all changes that are happening in the many repositories of the project. Each accepted PR must, as part of the CI process, include its changes into a Changeset file. All changes will be reflected on a static page that is updated each time a Changeset is modified. The result is a RSS-like feed of changes across all the project’s repositories.
This centralized feed isn’t deployed yet, but will be available at https://changelog.hackerexperience.com soon after we migrate from Github back to Phabricator. This migration, and many other internal changes, are expected to happen during December.
When you say “we are working on HE2”, who’s we?
You can’t hold talented people forever if you don’t have the money.
When I started the crowdfunding campaign, our goal was to gather USD 50,000 that would allow a team of 10 people, including me, to work on the project on specific areas like Web, Mobile, Terminal and Game Design.
When the campaign finished, we raised (free of taxes and fees) ~USD 11,000. While that is a considerable amount of money, it is nowhere near the amount required for the original team. That’s when we dropped the “Terminal” and “Mobile” versions, with a starting team of 5. After a few months the money was quickly fading out so we had to cut the “Game Design” team. We were now down to 4 members.
These 4 members — me, Allan, Charlotte and Rodrigo worked for almost two years together, in the first two versions that were thrown away. Somewhere down the road, but not all at once, the teams have changed. It was now me, Christian, Pedro and Max. This lasted for 18 months, give or take.
Right now, with basically no money, it’s me (>full-time), Max (part-time) and Lucas, a newly hired part-time designer. That’s “we”.
But enough about the past. How are things NOW?
Stressful. Deprived. Depressed. Struggling. But progressing!
- Helix, the server-side implementation of HE2, has ~70.000 lines of Elixir code (90k if you include documentation and comments), from which ~60% are actual game code and ~40% are tests.
- HEBorn, the client-side implementation of HE2, has ~45.000 lines of Elm code (55k if you include documentation and comments), from which ~95% are actual game code and ~5% are tests.
- Infra, the automated infrastructure of HE2, has ~5.000 lines of automation instructions (Ansible declarative YAML) and ~1.500 lines of code (Python, Shell Script).
Together, our front-end (HEBorn) and back-end (Helix) totals 115.000 lines of code, almost all of it written in the past 18 months. That’s actually a lot, considering ~90% of it was written by three full-time people.
In the software development industry, no one considers the total number of lines of code a useful metric for any purpose. However, we still do not have a playable version, and one could wonder what the heck have we been doing during this time.
Why don’t you have a playable version? What the heck have you been doing during this time?
“Playable” is quite an arbitrary term. Did you know that in our current code, most of the Legacy features are already implemented? In a much better way, of course. Plus, it’s fully tested!
You can hack other players, download/upload files, install viruses, collect them for money, edit/recover logs, exploit connections and overflow processes, create and use arbitrarily large bounces (VPNs) and much more.
And here’s an interesting fact: implementing these features were not the hardest part. Far from it. I’d venture a guess that these features account for 30% of our total work done so far. And you might wonder, “well, that’s pretty much all you have to do”.
Yes. And no. We should focus on features — and we are. But we also should do it TheRightWay. And that comes from experience — if you’ve ever played Legacy, you know what I’m talking about.
Legacy, written by me from 2012–14 in PHP, does not have a single test. It does not have any documentation. Its architecture was not planned at all. And the result is an unmaintainable spaghetti mess. It was a blessing experience, because I never again want to deal with code like this.
Remember about the “ideal, awesome” game I said we want to build? I don’t know if we will ever achieve this objective, but I do know that there is only ONE way that will happen, and that’s if we get the technical foundations of the game absolutely right. Well-tested code that you can trust. Well-written documentation that you can easily understand. But most importantly, a well-designed architecture that you can quickly and reliably expand upon.
And that’s why ~70% of our work have been towards building a framework, some sort of operating system that HE2 runs on. Let me give you three examples — two related to the game, and one related to the architecture. I’ll try not to get too technical.
Example 1: Process OS
The most challenging part to implement in the whole game are processes. Within Hacker Experience 2, a process is any action that takes some time and consumes some resource. Sometimes you know exactly when the process will finish (e.g. when downloading a file), sometimes not (e.g. when exploiting a connection or brute-forcing a password). Sometimes the process runs forever, either recursively (e.g. when using anymap or recovering logs) or as a daemon (e.g. firewall, password hasher, IDPS, tracer etc).
On top of that, processes must accurately and quickly allocate resources. This is a lot harder than it sounds. Want a wild example? Assume server Ahacked server Z, and Astarts a download of file f, which weighs 100 MB. Ahas 10 Mbit downlink, and Zhas 5 Mbit uplink. The download speed will be 5 Mbit (625 KBps), as it’s limited by Z’s uplink.
Now suppose the download has been going on for 5 seconds, when a random user U, with 2 Mbit downlink, also starts a download on server Z. Now, Z’s uplink must be shared between both Aand U. That means U’s download will go at a speed of 2 Mbps, and A’s download at 3 Mbps. But when that happens, we need to save the previous 5 seconds that downloaded 3.125 MB of the file.
Now suppose Ais actually connected to Zthrough a bounce with nintermediary hops. In that case, the available downlink and uplink will be determined by the bounce’s lowest downlink and uplink. Let’s assume that server Lhas the limiting downlink and uplink of 1 Mbps. Then, A’s download will go through at a rate of 1 Mbps.
Now suppose someone started a download at L. Now, L’s limiting uplink is 0.5 Mbps, and as such A’s process should be updated accordingly. However U’s process should also be updated, as now there’s more uplink of Z available for him.
Ouch. Got the gist? And this could easily snowball. Any change on process allocation (process was started, paused, completed, or had its priority altered) can easily affect hundreds of servers.
I could ignore this issue, but having a proper hardware resource allocation is key for the strategic mechanics of the game. Based on previous experience that was only possible due to the restarts we had earlier on, I’ve spent at least two full months working on a generic process scheduler that solved these problems. It is actually based on the Linux scheduler!
Hacker Experience 2’s current scheduler can correctly infer the ideal usage based on several factors, including whether the process is paused or not, whether it should use as much resource as possible or not, and whether it should use remote resources as well. It does so in a very quick way (it can schedule a server with 100 (!) processes in less than 1ms), so even if the snowballing scenario happens at extremely busy servers, it would take at most a few seconds to balance everything.
Now, to the most interesting feature. In-game events can send signals to existing processes, much like UNIX’s SIGTERM, SIGKILL, SIGHUPetc. This allows us to expand and add any game mechanics to our processes. Should a download process be deleted when the underlying file is deleted? Simply add a handler for the signal SIG_SRC_FILE_DELETEDsaying so. Should a exploit process take longer if the target connection gets encrypted? Simply handle the SIG_TGT_CONN_ENCRYPTEDsignal.
Example 2: Support for future mechanics
I plan to keep working on HE2 for years to come. It’s a hobby, and above all else a game I’m crafting to myself. If it succeeds and gives me money, great! That means I can work full-time on it. If not, that’s OK too! I’ll still work on it during my free time. It feels like gardening — and it’s a codebase I enjoy working on :)
That means I need to know very well all features that I want the game to have, even if we are talking about features that can only be added in 5 years (!) or more. Fortunately (or not), this is a game I’ve been dreaming with since 2006, so I have a pretty good idea of what I want it to have. And how to make it fun (I don’t think I’ve ever talked about the Clan mechanics, and the dynamics behind Clan Wars. But believe me, they are breathtaking).
One of the features that may be added in 5 years or so is support for hardware RAID. Other than serving the purpose to educate the player, this feature can increase the resiliency and availability of the server during a botnet-controlled attack (DDoS). But before that, a few months after the BETA but definitely before the “1.0” release, we’ll add support for USB disk (pen drive). How can we prepare the foundations for these features? Here’s what we did:
Files do not belong to a server, as one might think. And they do not belong to a HD either! Internally they belong to Storage. HDs have access to these Storages, and Servers have access to HDs. When the player opens his FileExplorer, we’ll get all mounted storages and return their files. Detached a HD from Server 1 and attached it to Server 2? That’s where your files will be now. Two HDs share the same storage? The file is replicated among them. In a way, this resembles LVM’s VG or ZFS’s ZPOOL.
Implementing the Storage abstraction did slow us down a bit, but the flexibility it brings is welcomed, and an absolute must for future features we have planned (including some in the near future, like USB disk). If we have not planned for this change, adding it would require a substantial refactor, to the point that it would not be worth the trouble.
Another example. You know the process scheduler I’ve proudly talked about a few paragraphs above? It was designed to be VERY, VERY EASY to add new hardware resources. Current resources are CPU (MHz), RAM (MB), DLK (Mbps), ULK (Mbps) and Time (s). Want to add HDD/SSD (IOPS) or RAM (MHz) or GPU (MHz)? You don’t have to change ANY of the scheduling logic; you don’t have to change ANY of the existing processes. All that needs to be implemented is how the new hardware resource should behave for operations like sum, sub, mul, div, overflow; and from which hardware component it comes from. That’s 100 lines tops. And if the resource behaves the same way as another one, the implementation can be shared, meaning that adding support for HDD/SSD (IOPS) requires less than 10 lines to be changed. It’s all done, all ready, all beautiful.
Example 3: Event-based architecture
Ever since we started working on this final attempt, one thing stood out in our architecture: events. We use it everywhere. Sometimes even for Request-Reply communication! This allowed us to grow a very decoupled and modular codebase. Working on HE2’s back-end is very easy because you rarely need to know anything else other than the domain you are working on.
Suppose I want to implement something like, say, a Keylogger. The Keylogger, once installed, will send me an in-game mail with every log created on the server (we may add Keyloggers in the future, but that’s definitely not how they work in real life). All I have to do is:
1. Subscribe to LogCreatedEvent (meaning that, every time a Log is created, a function will be called. It will receive the event as argument).
2. The event contains the Log. Get the server_id from it, see if there are any Keyloggers running on any of the attached Storages.
2.1. For each running Keylogger, emit the event KeyloggerSpottedLogEvent, containing the Log and the Keylogger id.
3. Subscribe to the KeyloggerSpottedLogEvent.
3.1. Create and insert on the Database the Email containing the log data
3.2. Emit EmailSentEvent, containing the email and the player who installed the keylogger. Make sure it implements the Publishable protocol.
The Publishable protocol is a global handler (subscribes automatically to all events). If the event implements the Publishable protocol, its contents will be rendered and then pushed to the client. Now it’s the front-end’s (HEBorn’s) job to display the email.
Notice how easy adding this mechanic was, and how little you had to know about a 70kloc codebase. You don’t need to know how processes work, or what must happen for a log to be created, or how the event is published to the client. All you had to do was extend on existing events, making sure that every time a log is created, you proceed with the logic flow described above. In other words, to create the Keylogger mechanic you only had to know about the Keylogger itself!
Why create the KeyloggerSpottedLogEvent? One could say this is overkill. Maybe it is. But suppose we want the Keylogger to notify the user not only about Logs that were created, but also about Logs that were modified? It’s simple. Just make sure to emit a KeyloggerSpottedLogEvent after a LogRevisedEventis emitted. That’s all. If you had implemented the Keylogger logic (step 3) directly as a reaction to LogCreatedEvent(step 2), you’d have to either repeat your logic or refactor the code (and the underlying tests).
Testing this feature is just as easy. The tests are discrete, they have a clear beginning and ending. As long as the other events are working properly (here, the LogCreatedEvent and the PublishableHandler), your implementation will work just fine.
Easy, right? That’s why we spend ~30% of our time working on features. Because we spent another ~70% designing and working on the architecture that allowed us to implement and extend the codebase in such fashion. And that’s the beauty of an event-based architecture. Extending a highly decoupled codebase is easy and enjoyable. Add to that a highly concurrent framework like Erlang/OTP or Akka, and you’ve got yourself ProgrammerZen!
But please, do not take from this that I think HE2 architecture is perfect. Far from it, there are several flaws in it, some larger than others, but the key takeaway here is that this architecture that we’ve built works for us, and it’s been easy to use, understand and extend based on our use-case. There is no silver bullet.
If you have all of that ready, why isn’t the game playable then?
As an aspiring entrepreneur, I hear all the time about how important an MVP (minimum viable product) is. How it enables a faster feedback loop and so on. And that’s very true! But creating a game MVP is not as easy as, say, a SaaS application.
A well-designed game has all its features in a “closed loop” balance system. Each feature has an impact on the game balance, and all of them must be present to allow for a balanced version of the game. Furthermore, all elements of the game contribute directly to the overall experience. That includes sounds, graphics, animations.
That’s why I’m hesitant to release an MVP — or Early Access — version of the game. Because the game simply does not have all its systems integrated yet, leading to a potentially unbalanced and unpleasant experience. That said, I promised a BETA version, and it will be released even if I don’t consider the game to be fully ready.
However, in order to release the BETA version, there is one major component that is required and isn’t ready yet: the visual.
You are saying you don’t want to release the BETA because it’s ugly?
Mostly. It’s not only about aesthetics, but also about the actual user experience.
In computer programming, it’s possible to create the underlying features without a view. That’s exactly what we’ve been doing: we have a good chunk of the raw gameplay ready, but it only exists in a testing environment (development tests).
When we achieved a set of basic features on the back-end (server-side), we started working on an initial version of the front-end (client-side). That’s when HEBorn was born.
Interestingly, even in the front-end we can develop focused on features, not the view. For over a year, we worked on the front-end until it reached feature parity with the back-end.
Finally, the client application, the one players will actually interact with, was ready. Sort of. It was ready in the sense that is supported most basic features, and even some non-basic ones.
For instance, the current front-end version supports connecting to multiple different players at the same time; changing your “Desktop view” to another player’s Desktop; switching gateway/remote contexts on the Desktop apps and so many other Desktop-related features that do not exist in Legacy.
But it wasn’t ready in the sense that we never made it visually pleasing. We were just concerned about the features.
There’s one additional problem, not related to aesthetics but to user/gameplay experience. As stated, most of the time we were concerned on implementing the features, since this is the most basic and essential requirement. But we spent very little time on making these features easy to use and intuitive.
In other words, not only our front-end is visually ugly, but you need to have insider knowledge to know how it works. That’s the two things we need to fix for the first BETA release. And we are working on them.
How exactly are you solving the design problem?
We started last September a major overhaul of HEBorn (the front-end) exactly to solve this design problem and finally release the BETA. We are working on two fronts.
First, and most importantly, we need pleasant aesthetics. No more ugly, black-and-white design like above. And for that we need a designer! That’s why Lucas is now part of our team.
We decided to work on HE2 visual in an iteration-based way. Right now we are going through the first design iteration, one that is designed based on how we think players will interact with the interface. Once we release the BETA, we’ll collect feedback and create the second iteration, based on how players actually use the interface. Finally, once we develop all features needed for the 1.0 (gold) release, we’ll add another iteration, now with the player feedback of all features.
I’m going to show you the first iteration. It’s not ready, there are several details that need some working on, but it may help give you an idea.
As I mentioned, there are some details that still need to be worked on, but we (me and Lucas) are more and more inclined to use the screenshots above as a foundation for the first design iteration.
If you have feedback with regards to the visuals above, it’s OK to send us an email. But do have in mind that once we start the BETA testing, we’ll be 100% focused on what YOU, the user, think of the design, and what should be changed/improved.
Lucas is also working on a bunch of animations meant to “give life” to the game interface. Unfortunately you’ll have to wait for the BETA release to see them.
I said we were working on two fronts. The other one is a major refactor of our front-end.
Please don’t tell me you are throwing out the existing front-end…
I am. But fear not, my friend. It’s not as scary as it sounds. Let me explain.
As I mentioned above, we threw out the first two implementations of the back-end until we reached one that was great for our purposes.
Never did we started a second implementation of the front-end, because it never got “far”, in the sense that we never had created a front-end that contained all the basic features of the game.
We do now. After ~18 months of work, we reached the current version of HEBorn. We started it with little idea of how the game would be used by the players. We faced several challenges, stuff we did not expect, things we did not think we’d face on the game interface. For each of these challenges, we found a solution. Some of these were reworked, always thinking on “how the user would use them in the game”.
The most memorable challenge was definitely the Desktop Context issue. By default a player is connected on her desktop. She can open apps and windows on her desktop. But the player can also switch to other desktops from people she hacked. How do we deal with windows? How do we know which window belongs to which desktop? How to do this in an intuitive, usable, easy way?
Right now we have a new mindset. We can create in ~2 months what it took us 18 months, because we know what we need now. Furthermore, it’s time to integrate the design Lucas is working on with the client, so this is the best moment to do that.
That’s why I firmly believe that starting afresh will be the best decision for the long-term health of the project. It will remove legacy and confusing logic that was created when we had no idea of how the interface was supposed to behave.
Why there have been little to no activity on Github for the past few months?
Ok, here’s when things get dark. Let me paint a picture for context:
- I’m working on Hacker Experience 2 for almost 4 years.
- On July/2016 I promised the game would be released by December/2016.
- There are hundreds of people who have paid to see this game.
- It’s almost 2019 and we haven’t released it yet.
- Some people think I ran away with the crowdfunding money.
- I actually have no money right now.
- I actually got indebted because of the game.
- My relatives are expecting this game as much as you are.
The thing is, I have no words to describe how much pressure I am under right now, hopefully the points above can give an idea. And that’s the tip of the iceberg; I don’t want to get too personal so I’ve omitted some facts.
This is not a matter of “Renato does not handle pressure well”. I do. I have been under several stressful situations before, but this one is different. It’s different because it’s not about me, it’s about thousands of backers who believed in me. It’s not about me losing money, it’s about other people who poured their money into the project.
Even if I release the game, even if the game is as great as it was advertised, even if all else happens exactly according to the plan, I still will think that I ultimately failed, because I made other people wait for 4 years.
This made me sick. Literally. I started writing this blog post in June. Ever since, I got the common cold at least one time each month that has passed.
Around March I started having burnout symptoms. This was my third burnout, all three happened during this project. The first two lasted one month each. This one is fading out now, in September, 7 months after it started.
Normally, I’ve been able to work for 10–12 hours per day, every day of the week, on Hacker Experience 2. It got reduced to 6 hours, then 4, until I was absolutely unable to work on the game.
(During this burnout period I was not able to work with development, but I was still able to enhance the game design and sort out several mechanics that were unfinished or immature. At the very least I managed to have some progress, so I don’t feel like it was 100% wasted time.)
Is this related to the lack of project updates?
This whole situation directly affects how I handle the project publicly. The delay, the pressure, the burnout create an unbearable situation where I am unable to talk to other people about the project. I know this is confusing (mostly because it’s something that happens within my mind), but let me try to explain using other words:
If I don’t have anything to show, i.e. if I can’t share a link with the playable version of the game, I don’t feel like I “deserve” to give an update on the project (because I know people are expecting the release and nothing else). This actually prohibits me from checking my email! I can’t read my email because I have nothing to show to you!
So I just went over 7 months without opening my email client. This got me into a lot of trouble, as I missed several deadlines of stuff unrelated to the project. I actually got officially dropped-out from my college, since I couldn’t watch classes while the game wasn’t finished. I got into legal trouble because I’ve missed several tax-related deadlines/invoices.
It then snowballs, because each month that I do not give an update, it’s one extra month on the “it’s been X months since your last update” pressure.
On top of that, I get the constant thought that “everyone hates me because of the delays”. In reality, I don’t know to what extent this is true, but it makes handling public relations almost impossible when I think everyone is out there to yell at me.
Ouch. Needless to say, this is absolutely unhealthy and it must stop now before it’s too late. I can’t let this situation permanently affect my health.
I’ve already made the decision of switching to another area after the game is done. At least for a while, I plan to stop programming (something I always loved). Maybe I’ll try something as systems administrator or security engineer, but I need some time off development.
HE2 is a project I love. It’s a passion for me. The last thing I expected from it was that it would make me dread computer programming, even if temporarily. And I still am not sure if it’s a temporary thing.
You said you wanted to talk about refunds?
Now let’s talk about money and refund. This is a delicate subject, so bear with me. Here’s how things started, roughly:
- I presented the outline of the project in a crowdfunding campaign. A project that all members, especially I, believe in.
- You, backer, found out about the project, got interested and, based on the proposed features and expected outcome, made a donation to support the development of the project.
- Ever since the crowdfunding campaign has ended (and before it, actually), me and my team have been working full-time on the project.
- (The teams may have changed in the meantime. But I’m the sole responsible for the crowdfunding campaign and I have never stopped working on the project).
- Then, some time has passed. The project wasn’t delivered in time (and, as of now, hasn’t been delivered yet). Updates about the project went silent. Based on the context, many of you believed we have gave up on the project (which is a reasonable conclusion, given the lack of updates).
- Naturally, many of you asked for a refund, since the outcome you expected (the game you paid for) hasn’t arrived into your computer and (apparently) no one was working on it anymore.
This is a very clear and truthful state of how things evolved. And I, backer, understand your feeling. It’s OK and even reasonable to be frustrated and disappointed.
Before we proceed, I have to apologize. I deeply apologize about the delay of the project. The original release date, and the many release dates I announced later on, were wrong. Very wrong.
I vastly underestimated the project complexity (which unfortunately is something very common in software development).
When I realized the sheer complexity of the project, I could have given up. I could have released a half backed, poorly designed game. But NO! This is my passion, my life’s project. This is a game YOU believe in, exactly (or better than) the one presented at the campaign/trailer. So I embraced the third option, which is to “suck it up and go all in”. And I went. And I am.
Back to the refund topic. During these 4 years I’ve been working on HE2 (2 and a half since the crowdfunding campaign), I always worked more-than-full-time on it. I never had the time to work as a regular employee for a big (or small) company. Meaning, I have not had any source of income for the past several years, other than the crowdfunding campaign.
I live a very simple life, possessing the bare minimum I need for work and nothing else. I have no social life, so I never get out of home. I live in a relatively inexpensive country (as far as living costs goes). And for the few hundred dollars I need to “survive”, my parents have always been there for me. They sponsor me because they, too, believe in the project. And trust me, they know I work hard on it, otherwise they wouldn’t lend me money at all.
“Need money? Get a job!” — Every parent in the world.
The money we raised during the crowdfunding campaign was used, 100% of it, to fund the game development. It is long gone now. It has been for over a year. Currently, like before the campaign started, my parents are the ones keeping the project alive.
This means I have no money. I have no money at all. And I can’t print money. But I can get a paying job. And I won’t get a job until I finish the game.
What does this means for refunds?
As I said above, I understand that many of you want a refund. It’s OK. And I WILL refund you. Except it can’t be done now.
What I need you all to understand is, I still am working on the game, and plan to keep doing so for the next several months.
If, after some time, I still haven’t finished the game, here’s what I’ll do:
- Get a regular paying job.
- Refund ALL of you, including the ones who never asked for a refund. Will take a few months.
- Keep working on the game on my spare time.
On the other hand, if the game is released within that time, I will:
- Use the game revenue to refund those who still want a refund. It’s OK to ask for a refund. The released game may not be what you had in mind. May take a few months.
- If revenue is small, I’ll get a regular paying job and work on the game on my spare time.
- If the game generates a meaningful revenue, I’ll keep working full-time on it, bringing improvements, updates, story expansions and new features.
In all of the possible scenarios, those who ask for a refund will get their money back. And if the game is never released, you have my word that all of you will get your money back.
That said, what I need right now is time.
What? More time?
Yes, I need more time. The game is not ready yet, but we have gone a long way from the beginning. The BETA version is not that far away.
I will not give up now. It means I will keep working on the game for the foreseeable future. It also means I cannot refund any of you right now, since I can’t get a regular paying job while working more-than-full-time on HE2.
So, I ask you, dear backer. You may not trust me after so many delays and quiet periods. But right now, I need you to trust me and give me some extra time. We are almost there.
How long?
Hard to say exactly. But I need to make up with you. The next few months I’ll work non-stop on finishing the Alpha version. Once released, you’ll have a glimpse of what we’ve been doing for all these years. And hopefully you will get hyped again for the game. I truly hope so.
My internal project planning says we are very close. As soon as I finish the internal structural changes I’ve been making on the front-end (might last one extra month), the “road to Alpha” will be a lot more clear and accurate.
As such, based on this planning and some other stuff, and also adding a little extra time because things never go as expected, the Alpha version is scheduled for early March, 2019. The BETA version, containing most of the game features presented on the campaign/trailer, early June, 2019.
I know you have already waited for so long. I understand it. But can you hold it for a few more months? This time we are very very close, and now isn’t the time to give up.
If by June, 2019 the game isn’t released yet, I’ll get a regular paying job and start the refund process. I will, then, keep working on the game, but on my spare time. If this scenario happens, I assure you the game will be released, but it will take much longer.
So, that’s it. Before finishing, let me make one thing clear: I am a person, not a robot. I have feelings, and for the most part these feelings affect how I work.
I need you all to know that I’m giving my life for this project. Heck, I’ve worked on it for more than 10 years, and I owe everything I know to the Hacker Experience franchise. The new version, HE2, has being a part of my life “only” for the past 4 years, but it’s something I have a deep passion for, and something I’ll keep working on until it’s over. And then I will work even more on it, for software is never over.
Again, I know and understand you are frustrated with the delay. I understand that the $10, or $15, or $60 or whatever amount you backed the project with, could be better spent on another game. Probably that other game would have already arrived at your computer.
But please, do not give up on Hacker Experience 2. If you backed the project, it’s because — at least for some time — you believed in its premise as much as I do. And whatever gets in our way, whatever happens during development, we want the same outcome: that game we originally cherished for. Or something better!
And I assure you, my friend. That game will be released.
UPDATE: December/2019
As I mentioned at the start of the post, even though I finished writing it in September, I only published it in December. The reason being the same I explained above about not being able to open my email client or publish game updates.
The fact that you are reading this post right now, however, is an immensely good news. It means my burnout is over (for the most part). It means I managed to publish an update about the project, so I just dropped half of the pressure I described above. I still haven’t pressed the Publish button but I already feel a lot better.
As it turns out, I have got great news about the project as well! I managed to work full-time on the project this last month (November). The last time this happened was in February IIRC!
Remember the front-end “major refactor” I told you about? It’s going smoothly. In ~1 month of part-time work, I’ve lifted most of the hard parts that HEBorn had, except in a much cleaner and direct logic (that’s not a critique to the quality of HEBorn or its developers. Instead, it’s a statement that the rewrite contains only the needed logic and nothing else. During the development of HEBorn, we didn’t know we wouldn’t need some parts, and that added unnecessary cruft and complexity to its code.)
During my “other part-time”, I worked on a major refactor of Helix, our back-end! I mentioned in the post above that I rewrite is not needed on Helix (and that’s true), but it still had a couple of rough edges in its architecture that could be improved. I did that, but I also went a bit further.
— Technical warning! —
I’ve decided to no longer use Websockets! This is a major decision, and one that may make little sense since we are developing a real-time game.
When we started working on HE2 in 2015, Websockets were the clear choice. HTTP/2 was still a draft.
We were aware of the downsides of Websockets (it’s stateful! (re)connections are expensive!), but we missed the fact that, by using Websockets, we wouldn’t be able to leverage most of the REST-based ecosystem. Most notably, we wouldn’t be able to have (cheap) DDoS protection from Cloudflare while using Websockets.
By now, HTTP/2 is solid and well supported. When coupled with SSE, an old HTML5 technology, we have almost the same benefits of Websockets (with the added weight of headers) while being able to leverage the whole HTTP/REST ecosystem!
Not only that, dropping Websockets greatly simplified our architecture, as the stateful Websocket nodes were merged with the stateless Helix nodes. (We still have stateful connections — SSE — but I’ve developed the code to handle this state in a stateless fashion). It’s confusing, but it works, keeping our infrastructure much more simplified.
Speaking of infrastructure. Also during the last month, I’ve added some cluster awareness on our back-end, meaning it now supports multiple Helix nodes running concurrently. Our intention is to have several small servers running Helix (instead of one big server), in order to increase resiliency and handle more traffic while keeping costs down.
Under sustained load, Helix* can handle 50 syncrequests per second on a single ARM-based server (a credit card sized computer). sync request is one of the heaviest requests the server has to handle, for it has to gather the entire game state of a given player, yielding dozens of read queries against the databases.
*(Do note that we are talking about Helix, the application server. The database server was on an SSD-backed PC).
Having an idea of the capacity of our back-end is important because, since Hacker Experience is within the hacker genre, we do suffer constant DDoS attacks. On Legacy, simple DoS attacks are enough to bring the entire system down. Helix has achieved a level of resiliency infinitely superior to Legacy’s.
Still related to resiliency and infrastructure, I’ve added support to requests rate-limit. It means we will no longer need those annoying CAPTCHAs while making sure the application is secure against some BOTs trying to take the system down.
The rate-limit logic also applies to logged-in players, so BOTs making several requests per second, or sustained requests across several minutes, will be flagged and blocked.
Not related to front-end or back-end, I’ve decided to move away from Github, back to Phabricator. First, because we favor a 100% open-source stack whenever possible, and Github isn’t open-source. Second and most importantly, I feel I’m more organized and productive in Phabricator. I still plan to keep a mirror of the repositories on Github.
Our infrastructure has changed so much since the first continuous deployment we had about ~1 year ago, that I feel I also need to rewrite most of our Infrastructure playbooks.
(I know what you are thinking. I am a big fan of rewrites. They allow me to create a code that is more direct and simpler than the previous one, for now I know exactly what will be needed and how things are supposed to behave).
For the infrastructure codebase, however, I also feel it’s time to part ways with Ansible. I will attempt to recreate our infrastructure automation logic using the SaltStack. Its event-driven management seems to be a better fit with the mindset of our project.
— End of technical warning —
I guess that’s it. A lot has happened in the last month, uh? Consider this to be a draft of the “Changelog” that I plan to centralize automatically. Albeit the Changelog will probably be more detailed (technically) and less prose-y.
It seems the post is ready. I’m actually very anxious to press that green Publishbutton. Maybe I’ll just procrastinate a bit more and write a Q&A section.
Quick Questions & Answers
What happened to Leaks? Why are you posting on Medium?
As part of our, ehr, “budget constraint”, I’m no longer able to afford the servers that kept Leaks and some other stuff online. That’s why Leaks is offline, and it’s also why I’m posting on Medium. Recreating Leaks is not a priority for now. But sometime in the future we’ll definitely want a dedicated blog for Hacker Experience, and honestly I’m not a big fan of Medium.
Are you aware of the broken links on the project homepage?
Yes. They are a direct result of the “budget constraint” mentioned above. Currently I don’t have the time to fix them. Gotta release the game first.
I have called the phone listed on the homepage and no one answers. Why is that?
Sorry, also as a result of the “budget constraint” I no longer pay for that phone service.
I have sent an email ages ago asking something related to the project and never got a response.
Ouch, I’m deeply sorry about this. The reasoning is explained on the post, although, just like the lack of updates, it’s an inexcusable situation. I have created an empty inbox and am trying to keep that one tidy, answering all incoming e-mails. Please, “BUMP” your email so I can get notified on the new inbox and get back to you :)
When will we see you again on the Discord server?
I plan on coming back there soon. First I have to get rid of the whole anxiety I described above. Then, I have to flush out of my system the “everyone hates me because of the delays” thinking. I can’t come back while I still have that thought in my mind, it’s not healthy. I hope you can understand it.
I recently found out about the game. How do I buy a copy?
I’ve halted orders for now, and will only accept new ones when the BETA is released. It helps with the whole “pressure” thing. Hope you can understand.
What’s the difference between Alpha and BETA?
BETA is the first “truly” playable version of the game, with most of the features we presented during the crowdfunding campaign (and a lot more, actually, because most of the original mechanics evolved during these years).
Alpha is a step before BETA. It is highly experimental, meaning server data may be reset a few times. It’s intended to be the first real interaction between the game and the players, with the main goal of allowing BETA to be a more stable version. It also allows you to have an early interaction with the game mechanics. You will be able to start hacking, installing viruses, managing your botnet, launching DDoS attacks and much more as soon as Alpha is out!
Both Alpha and BETA are intended to be an experiment on how things will look like in the “final” version (1.0, a.k.a. Gold). And that’s the perfect time for you to give some feedback. Tell us what you liked, didn’t like, what you would do differently and so on.
How do I start contributing to the project?
Code, documentation, translation, gameplay testing, feature suggestions, graphic design feedback. There are many different ways people can contribute to the project, but currently there’s no documentation on how to start contributing. I’ve tried to fix this with the Contributor Task Force (CTF), but unfortunately it’s something I can’t keep up while also being responsible for the back-end, front-end, infrastructure and game design.
For now, please hold on to that thought. Currently there is no easy way to contribute to the project (other than figuring out on your own by reading the code). I will make this a priority as soon as Alpha is released.
How do we know if you are working on the project?
Other than the usual activity at Github and the Changelog mentioned above, there’s one other interesting way. I used to stream while coding. It’s not a fancy streaming where you can see me. It doesn’t have video nor sound; it’s a simple mirror of the contents on my IDE buffer, hosted at floobits.com. When I’m developing, I’ll be streaming at renatomassaro/helix when working with Helix, and renatomassaro/heview when working with Heview (the new front-end). I’ll make sure to start streaming again from now on.
Is there anything you regret about the project?
Yes. I’ve made only one virtual friend ever since the project started, and we won’t talk anymore, mostly because of my anxiety issues described above. The journey is supposed to be fun. It’s not fun if you don’t make friends along the way.
You said that new features were “evolved” during the development. Care to elaborate?
Yes. There are many features that, I believe, most of you are not aware of, as they were designed after the campaign has ended. Some of them are:
- Botnet-based mechanics. DDoS is not the only action performed by your botnets.
- IDPS (Intrusion Detection and Prevention System). Automatically starts a trace connection when an attack is detected, among other things.
- Stateful firewalls. Block connections, blacklist IPs and filter DDoS attacks.
- PvP missions on Deep Web.
- New class of software — SaaS software. Has pros and cons over regular software.
- Software licenses: Open Source permissive (MIT, BSD), Open Source Copyleft (GPL), Copyright commercial, Copyright non-commercial. Further usage and development of these software depend on the license chosen by the software author.
- Source code from software you developed is hosted at BitHub :).
- WiFi (enables location-based mechanics on Desktop).
- Logless remote access (through Exploits).
- Exploits everywhere. Each exploited connection has a custom behavior related to its connection type.
- Three classes of hackers: Salarymen, Anarchists and Freelancers (names not final). Yes, it’s a cyberpunk plot.
- Corporations (clans) with custom mechanics and bonuses that apply to all members.
- Corporations may hire Freelancers during a Corporation War.
- LANs, Routers and Switches. Network topology planning.
- Single-player storyline (requires Internet connection).
- Side-channel cache-based attacks.
- Remote Desktop Protocol. Live view of remote player’s desktop. You see what they see (protected/sensitive windows, like database, player settings or chat windows, are censored and not viewable by anyone else).
- Streaming mode (sensitive information and IPs are automatically censored).
- Fully integrated HEmacs (VIM-like keybindings on the Desktop version).
- Integrated in-game Discord channel (pending Discord approval).
- Integrated in-game forum.
Of course there’s much more. These are just some features that popped up on my mind. Some of them are already implemented, some will be worked on only after the Gold release.
There were rumors that you would release the game for free. Is it true?
I really would like to do so, but I need to have a source of income so I can further develop the game (and possibly even hire people to do so).
That said, I am planning on releasing the single-player storyline (campaign mode) for free. It has relatively low server costs (when compared to the multi-player mode) so it’s probably OK to release it for free.
It’s either that or a pay-to-win game. I think we all want the former :)
One can always set up a local server and play for free with their friends. After the Gold release, making the local server installation as painless as possible will be one of my priorities.
How’s Phoebe?
