Launch-week catastrophe, the impact, and the recovery

The issue

This is not the kind of launch-week post I’d like to make. I received reports this morning (Friday, September 27th, 2019) that two users were having difficulties logging in. I asked for details. User A was logging in as User B somehow. This is obviously an incredibly severe issue whose bounds I needed to understand immediately.

Background / investigation

On Tuesday, September 24th, Bot Land launched. I had written a script to soft-reset people’s accounts, meaning they could keep their username, email, password, and superficial details, but not their items, in-game progress, etc. I tested this multiple times on PII-sanitized copies of the production database and everything went well.

On Launch Day though, everything did not go well. The script took ~10 times longer than it should have before I assumed that there was an unrecoverable error (I never actually verified whether this was true because I was deploying while live-streaming and I was already frantic to get the game online).

Instead, I rolled the database back about 3 days to when everything was working, reran my scripts, and Bot Land was up and running. Unfortunately for me, a catastrophic issue lurked beneath this rollback.

In Bot Land, users type their username and password and the client sends them over HTTPS to the server. The server salts and hashes your password, verifies it against what’s in the database, then returns an authentication token so that your username and password don’t have to be stored on the client. This token is a JWT, meaning it can be examined by anybody but only changed by someone with the password (i.e. me). The contents of a Bot Land token look roughly like this:

{
  id: 1234
}

That’s it. It’s just a user ID. Remember though, users can’t tamper with the payload without knowing the secret, so there’s no risk of a user specifying someone else’s ID in here.

User IDs in Bot Land are incrementing integers, meaning they will always be assigned in the order 1, 2, 3, and so on. For a game, it’s fine if you know that user ID #61306 exists; it doesn’t provide any personal information about that user.

So far, everything I’ve described is sound. Sure, you may argue that user IDs should have been UUIDs by default, but there isn’t a major problem yet.

The issue came with the database rollback. Suppose we had 5000 users before the rollback. Imagine a user whose ID is, say, 4993. Let’s say their username is Bob. Bob sets up autologin before the rollback, meaning when they get their token with {id: 4993} in it, they’ll store it locally for use later. After the rollback, an account with the ID 4993 may no longer exist, but Bob’s token representing that account is still in his browser somewhere.

When Bob loads Bot Land after launch, that token will automatically be provided to the server, only now it won’t refer to Bob. Bob has now logged into Carol’s account.

Impact

First of all, in this scenario, Bob would only be able to log in as Carol, a seemingly randomly selected user, but one that is never selected a second time, meaning Bob can’t log in as David afterward.

What information can Bob get about Carol? It’s only the information that Bot Land exposes. Ignoring the game information (like your blueprints, scripts, items, etc.), the only personal information that they’d have access to is Carol’s email address—not her password, payment information (which we never store anyway), or any other identifying information. Furthermore, Bob would only have access to that if he sought it out in the interface, e.g. via the “change email address” function in the game or attempting to verify the email address of the account.

Next, in order for Bob to do this, they needed to have autologin enabled during the 3-day period between the rollback and launch, then they needed to try logging in from the same device/setup during the 3-day period between launch and now. There are 106 accounts that had logins during both time periods, but autologin is a client-side setting, so we can’t tell how many accounts that number is narrowed down to. However, 36 of those were guest accounts, meaning there was no personal information on them whatsoever, so the maximum number of potentially leaked email addresses is 70. Removing spam accounts with obviously fake email addresses (like “asdhjkwhejkwhetkj@ewuitwetyuiw.com”), this brings it down to 59.

Furthermore, while the email address would have to be sent from the server to the client, we can’t know whether the client actually viewed the email address from any of the 59 potentially affected accounts.

Thus, it’s impossible to know exactly who was affected, but again, at most, your email address could have been seen by exactly one user. I will be reaching out to all 59 users with these details and an offer to restore some of their game progress.

Recovery

  • I deleted all 106 potentially affected accounts, including guests and spam accounts, meaning that there is no longer an issue with anyone’s email address leaking.
  • I updated the signing key for all JWTs so that any already-issued tokens will no longer be considered valid. This will require all users to type their credentials again the next time they log in.
  • I will update the signing key again in the future for any rollbacks that have to be done. Since Beta launched on November 7th, 2017, there has been one database reset in late 2018*, then the rollback and soft-reset that we did on Launch Day, September 24th, 2019. Rollbacks are almost always the result of some other catastrophic issue, meaning I can’t foresee future rollbacks, but they could happen.

*This particular reset was not susceptible to the issue described herein because autologin didn’t exist until 3 months later.

Thoughts

Bot Land has a total of two developers. It has automated tests. It has nearly 100k lines of code. We do our best to catch everything, but when something like this is found, I want to be as transparent as possible about what happened and what was done to fix this.

I apologize for the downtime, the lost game progress, and any other repercussions of leaking email addresses.

Growing pains

TL;DR

I’m putting this summary at the top for people who get linked here from the stream and don’t want to read the whole post:

  • Due to the growth of the stream, I need to cut back on how long I spend conversing with chat so that I can focus more on development.
  • The biggest thing you can do to help is to post game feedback and suggestions on the Bot Land GitHub. If you want to go the extra mile, you can first check to see if your feedback or suggestion already exists, but don’t let that step stop you from posting there.
  • I still want to be as interactive as possible without cutting into the development process too much.
  • Please read the rest of the post for the full explanation!

The post itself

My stream has been growing recently:

Stream growth chart

I’m by no means a big streamer and it’s not a goal of mine to become one. Instead, streaming is all about making Bot Land as good as it can be (as discussed in this post). I used to be able to thank each individual viewer as they followed me. Now, I find myself hardly able to devote more than a “hey <viewer>” to each person who chats. It feels less personal to me, but when tens of people greet you at random points in the day, you can’t deviate from your current mental context too much or you risk a big time-loss recovering your train of thought.

I always try to remind myself: “something’s gotta give”. I’m at critical mass right now with the stress of development, and it’s been bleeding into my life outside of the stream. I need to shift some of the weight of interactivity to viewers and moderators, which means I need your help!

Duplication

When it comes to the stream, I learned long ago that I would keep getting the same kinds of questions: “what are you doing on stream”, “are you standing”, etc. This prompted me to make commands to answer these, e.g. !botland and !desk. Eventually, there was so much chat spam from just running these commands that I made an FAQ page to consolidate the list.

That worked fine for basic questions; viewers are generally okay with looking through a single page for the answers to their questions. However, now that the beta of Bot Land has been released, the feedback flood gates have been opened—I’m getting bug reports, overall feedback, and feature suggestions in droves. This is great and I welcome it, but as mentioned, it’s detracting from the stream for me to respond to everything in real-time.

The plan

I would like to collect feedback about the game on the GitHub issues page. This includes bugs and suggestions. Ideally, I’d love it if you could do some quick searches to see if your issue already exists. If it does, please comment on it if you feel you have something to add. Regardless, feel free to file an issue even if you have a hunch that it was already submitted; it’s better to have duplicates than to lose track of a problem!

With that said, if you are filing a bug, please follow standard guidelines, the most important being: include as much information as possible (screenshots, steps to reproduce, any errors in the console).

Use your discretion when it comes to typing game feedback in chat. For example, if you run into a game-breaking bug whose error message says, “this definitely shouldn’t happen! Tell Adam immediately”, then add it to chat. Likewise with “passive” feedback like “cool game” or “I showed this to a friend”—those kinds of things are fine.

Beyond the plan to use GitHub more, I will also likely need to gloss over messages that I previously would have responded to. For example, people sometimes ask about choosing between text editors, computer hardware, etc. You can still discuss that in chat, but I probably won’t reply beyond acknowledging that I can’t help. If I get enough of those kinds of questions, I’ll try to add them to the FAQ so that you can still hear my opinion without me needing to respond individually each time.

Please understand if have to say something like, “this isn’t productive for me to talk about more than I already have”. It’s not a slight against you. Also, if I seem to ignore your message without any acknowledgment, feel free to tag me until I notice it.

do care about interacting with viewers/chatters, but I feel like developing Bot Land is my one shot at achieving my dreams and I don’t want to pass this up!

Redesigning Bot Land

My last post on this blog was nearly a year ago; I instead put my time toward recap videos. Now that Bot Land has been in development for a little over a year, I want to consolidate some the history behind its design!

The original design

Bot Land has always been an online strategy game with a focus on automation. However, our initial plan was for it to be a realtime player-vs-player game. This video sums up the gameplay. You make bots, write scripts, then throw them into the Arena to battle—sounds great! There was a big problem though: the strategy aspect was lacking.

When talking about strategy games, it’s important to be able to describe a player’s decision tree. A decision tree is formed around the rules of the game (“what can I do?”) combined with how you think about the state of the game (“what should I do?”).

In the original design of Bot Land, your main action consisted of choosing which bot you want to deploy. This was really two “sub-actions”:

  • Choosing which hardware your bot has (e.g. Missiles and Cloaking)
  • Choosing which script your bot has (e.g. only move while cloaked)

Bot hardware roughly follows a rock-paper-scissors model: Reflect will beat Missiles, but it will lose to Melee. The problem we faced with choosing a bot was that this wasn’t a meaningful decision. You couldn’t control where your bots were deployed, and the bots themselves didn’t have any knowledge of an enemy’s hardware. This meant that you may want to deploy a Reflect bot to counter the enemy’s Missiles bot, but you couldn’t guarantee that the two would meet up in the Arena.

For a minute, let’s consider a world where you could guarantee that your Rock bots encountered the enemy’s Scissors bots. The problem is that it’s not a difficult decision to make. Games like StarCraft that feature rock-paper-scissors mechanics aren’t as straightforward because it involves a lot of speed and dexterity to be able to pull off a particular strategy. In Bot Land, after you deploy your bot, your script takes over for you.

Without meaningful decisions to make in a game, there won’t be a high skill ceiling. That means that “pro” players won’t be much better than new players. This undermines a strategy game in my opinion. More importantly, playtesting of Bot Land was confirming our suspicions that the game wouldn’t be fun if we stuck with this design. This meant we had to revamp the decisions that a player could make.

Flux

I published this video at the end of July. It covers the section you just read above, but it also talks about how we planned on pivoting our game design. In short, we wanted to expand the player’s decision tree. Over the course of August, we tested concepts like:

  • Being able to choose a bot’s spawn location
  • Being able to control bots that were already deployed (via waypoints)
  • Building factories which would lock you in to a particular hardware loadout
  • Creating hero bots which could be teleported at will
  • Spawning scrap piles on the map to give players points of interest to contest
  • Having territory for players to generate money
  • Upgrading bots

In the end, we determined that the game still wasn’t very strategic. What’s more is that it was hard to tell if we were even closer to a solution. Maybe another week of prototyping would get us there, or maybe we were pursuing something that was impossible. I was the most stressed-out that I think I’ve ever been in my life. I was beginning to think that Bot Land was unraveling and that I wouldn’t even finish the project.

Right around this time is when Ian, the game designer, stepped down due to time constraints. He left the company on amicable terms.

I started to question all of our original assumptions about the game. I created a document while streaming and started brainstorming (it’s now 70 pages and largely unreadable, but feel free to check it out). Here are the most important conclusions:

  • If you’re expected to change scripts in a game, then the game shouldn’t be in realtime. There are two reasons for this:
    • It’s neither easy nor fun for most people to code quickly. It’s especially unfun on a mobile device.
    • A big problem with being expected to make major changes to automation while it’s running is that it means that the automation needs to run slowly enough for you to be able to identify where you need to make a change, but it needs to run quickly enough for you to know whether your change had any impact.
  • If you can’t immediately tell exactly how something is going to play out in an automation game, then the game should give you multiple attempts at winning. The reason software has bugs is because people make mistakes. I think it’s okay to punish certain mistakes in games, but I think it would be frustrating to lose in Bot Land just because you couldn’t accurately predict how several medium-sized programs would play out.

New design

I started the last two sections with videos, so here’s another one that covers the last section and some new stuff (hey, I mentioned at the beginning of this post that I’m just consolidating information here!).

The biggest change in the new design is that Bot Land will no longer be a realtime game. It’s still a strategy game involving automation, and you’re still playing against other people, it’s just that those people don’t happen to be present when you’re challenging them. This is very similar to games like Clash of Clans or King of Thieves.

As a player of Bot Land, you have two roles. As a defender, you set up your own Arena ahead of time. Your goal is to protect your CPU. As an attacker, you are matched against another player’s defense (and remember, the defender isn’t actually present during the game). You get three attempts (AKA phases) to take out their CPU.

In this new design, attackers take their time positioning their bots, choosing hardware, and rewriting scripts. When they’re ready, they can start the phase and watch their automation play out. Any damage that the defender sustains will persist throughout later phases. After the phase is over, the attacker can make adjustments to their strategy and pick up where they left off.

Because attackers get to react to what they’re seeing in the Arena, defenders need some kind of advantage too. Defenders are able to place more bots than attackers can, and they also get to install chips into their Arena. Chips can boost the damage of their bots, make them move faster, or perhaps even alter the rules of engagement (maybe one such chip will make your CPU immune to damage until the chip is destroyed).

After an attacker is finished, the defender will be given a replay of what happened. Maybe the attacker teleported over all of your bots and gunned down the CPU quickly, or maybe they whittled you down with fiery weapons. The defender can use this knowledge to patch up any holes in their defense.

Bringing this back to the decision tree of Bot Land, attackers now have these decisions:

  • High-level strategy
    • Do I want to put down many bots in one phase, or do I want to spread them out evenly over all phases?
    • Do I go after their chips, their bots, or their CPU?
  • Low-level strategy
    • Where should I put my bots?
    • What should their hardware be?
    • What should their behavior be?

It’s my goal as a game designer to give defenders all sorts of different tools so that attackers aren’t falling into the same patterns. It would be a shame if attackers could win most matches with the same strategy, so it’s your job as a defender to come up with something unique! There will be plenty of hardware and chips at your disposal, and of course you can always write custom scripts that will surprise attackers.

Conclusions

This new design of defenders and attackers is analogous to puzzle-makers and puzzle-solvers. Sure, there are going to be some dead-simple defenses that you come across and “solve” instantaneously, but my hope is that defenders will keep evolving them to the point where even seasoned players are challenged. It’s in this way that I think the new design comes together; a player’s ingenuity is what should decide how well they do in Bot Land.

I prototyped and playtested enough of the new design to the point where I can say that it’s worth pursuing. I’m sure there will still be tweaks down the road, but as long as they’re not major overhauls, I think we’re on track for a fun game!

Coding in JavaScript in Bot Land

(note: I sum up most of this post in the video that I’ve created below)

[youtube https://www.youtube.com/watch?v=uMlecpQi6pM]

The core gameplay mechanic of Bot Land™ is coding, so you may think that you have to be a programmer to play the game. However, one of our key tenets is to make Bot Land accessible to everyone—coders and non-coders alike! If you don’t know how to code, you can either use our visual interface, Blockly, or you can make use of Quick Scripts and Quick Snippets in JavaScript.

Quick Scripts are complete scripts that have been written for you, so you don’t need to understand a drop of JavaScript to be able to use them. You can load them up just like in the video, but you’re free to make any changes you’d like after that.

Quick Snippets are small blocks of code that you’ll find yourself using a lot (e.g. the “if (willLasersHit()) {fireLasers();}” code). They allow you to code without having to weed through documentation to find the API you want.

That’s it for now! If you want to check out the development recap video from last week, you can find it here:

[youtube https://www.youtube.com/watch?v=OgD53lz7gqA]

Twitch, here I come!

My only foray into Twitch streaming was when I wrote my own version (now and forever offline) of Twitch Plays Pokémon. That project was a lot of fun, but I didn’t get to interface with any viewers. Heck, I didn’t even have a webcam running. How am I ever going to achieve my goal of Twitch fame if no one on the Internet knows who I am?

Well that’s all about to change…! The world will watch as my view-count soars into the double digits. That’s right, forget about PewDiePie and Kripp, Adam13531 (update from 2022: now “AdamLearnsLive”) is about to have upwards of 10 viewers!

Here’s the plan:

  • I’ll be streaming the development of Bot Land™. The majority of the code will likely be written while I’m on Twitch. The only work that I’ve already finished pertains to prototyping and research, so you get to see me produce the very first lines of real code!
  • I plan on streaming for four days a week. I’ll post a schedule on my Twitch page once I’ve got some consistency.
  • I’ll start mid-morning on Tuesday, 9/8/15 (roughly 10:00 AM PDT).
  • If you want to support me, you can do so simply by tuning in or following me. If you’re feeling extra generous then you could pass the link on to a friend or two.

I’m super excited for this! I’m looking forward to having a “workplace” again despite being at home all day. I also love sharing knowledge and tips, so feel free to ask me lots of questions. And I’ve got a couple of fun things in store for viewers…

Remember: mid-morning PDT on Tuesday, 9/8: https://www.twitch.tv/AdamLearnsLive. Be there!

Bot Land: Origins – Part I

I came up with the original Bot Land™ almost exactly five years ago, but it never saw the light of day. Neither did the subsequent iterations that took place over the years to follow. In this short post, I’ll share some details about the very first version of Bot Land that I ever made.

If I recall correctly, Adam-from-five-years-ago wanted to prototype a game that involved programming, but he didn’t actually want to think much about what that entailed. He opened up SciTE (this was before the days of Sublime) and hammered out 3,000 lines of Python into a single file. He was truly a visionary of terrible code.

I’ll let Adam-from-2015 tell you more about it here:

[youtube https://www.youtube.com/watch?v=moS4jt2_UxU]

There are plenty of things that I want to explain about this video, but they won’t make sense until I cover the other iterations of Bot Land*. After that, I can finally tie all of them together in a post explaining how we arrived at our game design for the final Bot Land, that way you get to see the full evolution of the idea.

*Just a note: it will be at least a few weeks before we continue with Part II of this series.

Icebreaker

In Toastmasters, the first speech you’re supposed to give is the Icebreaker. There are two primary purposes to this speech: to introduce yourself to your audience, and to help determine your strengths and weaknesses as a speaker. I’ve always enjoyed listening to Icebreakers because of the connection I feel with the speaker by the end of it. That’s why I’ll be starting this blog in the same way; I’ll tell you a bit about myself, the company, and what you can expect out of the blog.

My name is Adam Damiano, and I’m a software developer with a passion for games. When I was a kid, I remember getting a TI-83+ calculator and being delighted to find a simple programming language on it: TI-BASIC. I spent hours making all sorts of games, sharing them with my classmates, and trying to learn more about the wonderful world of programming. I eventually removed the training wheels and graduated to their Z80 assembly language. After that, I got insight into more modern programming techniques through high school and college.

Many years have passed since then, but not much has changed. I still spend too many hours designing, programming, and thinking about games. With the advent of Twitch.tv, I can now spend more time watching people play games. The best feeling I have as a game developer is when I can see someone enjoying a game that I’ve made. That feeling is what’s fueled my dream of owning my own game company someday.

Thankfully, “someday” arrived sooner than I expected. About a year ago at my last job, I met a guy named Ian Smith. Our conversations always seemed to steer themselves toward game design; we’d bounce ideas off each other, and I would sometimes code up a prototype (remind me to write a blog post about “Path to VP” at some point!).

One day, Ian proposed just a nugget of an idea for a game: “what if we made a game where programming came into play?” I was immediately hooked. This is something that I’d been toying with for almost five years. In fact, the last game I publicly released, OpHog (GitHub page), heavily featured autonomy despite not exactly having programming elements.

The only problem was that this new game would take too long to complete if I could only put my post-work hours into it, especially since I was tired for most of those hours. After a couple of weeks of deliberation, I decided to quit my job and pursue this venture full-time. Ian and I are starting Xtonomous LLC, a company devoted to making quality games.

There’s not too much to say about the company itself yet. We’re based in Seattle. We only have two co-founders. Our first game will be Bot Land, which coincidentally was the working title for the four iterations of the game-with-programming idea I’d worked on before meeting Ian. For Bot Land, I’ll be doing all the coding, and Ian will primarily be handling the creative side: game design and overseeing the user-experience (UX).

As promised, the last thing I want to tell you about is this blog. I plan on making at least one post per week. The content will almost always be related to Xtonomous, our games, or gaming/technology in general. Likely future topics include:

  • Technology choices and rationale
  • Details about Bot Land itself
  • Other games mentioned in this post
    • Those “four iterations” of Bot Land
    • Path to VP
  • The process of how Ian and I design portions of the game
  • Development updates with screenshots and/or videos

If there’s something in particular you’d like to see us write about, just let us know!