Shenglong's Blog

The Tale of Eternity: Part 3

In This Series...

If you only care about the DDoS attack, scroll down.

I remember reading Paul Graham's essays, which kept stressing the importance of finding compatible co-founders. One or two of them also mentioned that poor founder relations are often a reason for startup dying. All praise the oracle.

The Golden Years

EternityRO started with a bang. Our IRC channel was flooded with regulars, and frequented by random players here and there who needed support. The work never seemed to stop. We barely slept, and implicitly divided up our availability so there was always at least one person taking charge. Our dedication paid off, and Eternity Version 1 grew to about 1,200 simultaneous players. Boom, bang--whatever you want to call it, we really couldn't have dreamed for more. I was even scared we might go over our allocated 1TB of bandwidth for the month.

I didn't have Google Analytics enabled at this point, so here's the awstats from LiquidWeb (most helpful and flexible webhost I've ever worked with). These numbers aren't totally representative of the population, since this is for the website and forum, where most players don't go. I have actual game client stats which I'll post later... that's mind-blowing.

The Dilemma

Alas, all good things must come to an end. After several weeks of operation, the stress etched away Ayumi's sanity, and I witnessed a cascade of poor decisions. From freaking out at players to disappearing randomly, it was obvious that something was wrong. Like any friend, I tried to talk to her - but I was met with hostility and anger.

If this were a smaller project, I would've said, "Screw the project. You're stressed out and overreacting, and I want to help. Sit down." That's what any good friend would have done. But thousands of people were depending on us, and by this time, almost two thousand dollars had already been invested by the population. I had to weigh an emotionally unstable co-founder against a semi-business, and the enjoyment of thousands of people.

Enter Snow and Aisha - two of the five women responsible for EternityRO's booming success. Aisha's experience and personality made her well suited for management, and Snow was a development powerhouse. After trying to lure them into Eternity for a few days, they offered their assistance... but at a price. They wanted me to fire Ayumi and Prodigy.

The Decision

How does one co-founder fire the others? I guess you don't - but there was no equity in this case. After deliberating and arguing with myself for a night, I begrudgingly agreed. In retrospect, I can say I made the right decision. While it may sound morally abhorent, it might be better to look at this through consequentialist lenses. The alternative would've been to let Eternity die. Hey - I saw Social Network! I'm not the only one who did this!

Removing influential community members is usually a delicate process. You need to have finesse, and yet generate enough momentum to carry the motion through. It's actually remarkably similar to the Needle Through Glass technique I learned. The trick to using a needle to penetrate a pane of glass involves throwing the needle straight, yet with enough force to break the glass. A fault on either end will either bounce the needle back or shatter the glass. I'm getting a little bit off topic; I'll explain in another post some time how this and other techniques like breaking bricks and metal bars works. For now, here's a picture from when I was learning:

I changed the passwords on every system, and even the ssh ports. I closed down their forum accounts, disabled their SVN access, and redirected their emails to a new support email I set up - all while they were asleep. The last thing I needed was an emotional overreaction. Eternity was going strong, and I wasn't going to let it break apart from the inside. I announced their departures as mutual, and announced Snow and Aisha shortly after. The community didn't care all that much.

The Hiring Process

Ayumi and the others freaked and eventually disappeared from the internet (not an easy task). To compensate for the missing manpower, I charged Aisha with recruiting new support staff. She solicited applications, and to my surprise, recruited 20 new support GMs (Game Masters). Twenty is an exorbitant amount of people to introduce at once - but my objections were moot. Although Aisha was actually very small, cute, and huggable in real life, she scared me online.

Here's an approximate breakup of what happened to the twenty keen recruits:

10 of them quit or stopped logging in after the first month due to stress, incompetence, and other factors
5 of them dropped out between the first and second month due to stress, and real life
2 of them dropped out between the second month and fourth month
1 of them was fired for outright cheating
1 of them was fired for conspiring with enemies/competitors
1 of them stayed on and proved to be both exceptionally useful and intelligent. Hello Griffin!

Training twenty people at once was downright painful, especially since they were all volunteers. We didn't pay any of our staff, so you might be curious why anyone would work for us. Well, people like to have a sense of power, and that's really why people become Game Masters - even though they'll tell you it's because they want to help the community.

So why did 85% (17/20) of our new recruits drop out? Some of you HR fanatics have probably already decided that we had a bad selection or orientation process. While this might be true of this specific incident, our future open-hiring showed similar patterns. No, the problem is deeper rooted than our practices.

To understand why we had such a ridiculous churn rate, we need to look at motivation. People are only motivated by power for so long; once the honeymoon rush is gone, they lose their incentive. Add constant player-abuse, cheating accusations, and pressure to perform - and all of a sudden the job isn't so appealing anymore. Since they don't have much to lose, most people either quit or simply disappear.

The Irony

However, there was one out of the 20 who stayed on board, firm and committed. In fact, after version 1, I gave him administrative powers and started sharing all my plans with him. He essentially took on the role of co-founder, and at times even put me in my place. But why didn't he quit? Why didn't he lose motivation? You could say it was because he loved the community, but I have a better answer.

After version 3 died, we had a reminiscent chat. Apparently the only reason he applied for a Game Master position, was so he could cheat - but not in the classic way. Because all Game Masters could see which other Game Masters were online, Griffin used this to his advantage. He ran a small bot network on the server, and linked the software to his Game Master account. His script worked something like this (pseudo code):

if(GameMasters.Online.Count() > 0) { AllBots.logoff();}

Basically, because his bots avoided the inherent cheat protection, a Game Master was required to catch them. If they were never logged on when Game Masters were online, no one could ever prove he was cheating. I laughed when I found out. Obviously, Griffin stopped doing this eventually, and probably stayed on for the same reason as I did: intellectual curiosity. Several of our other staff members were also previously cheaters - ha.

The DDoS Attack

Recently, a wise man told me "when you run a successful tech startup, there are two truths: you will be DDoS'd and you will be sued." One night, I was sitting peacefully at my computer, when the server froze. After frantically trying to SSH into my box after being pelted with close to a hundred calls/texts/emails/forum PMs/IRC messages, I decided to log into my control panel to check my bandwidth. Oh my f***. Here's what I saw:

Normal traffic, then boom - a DDoS attack. Granted, I was on a 10mbps line, but I never figured I would need more than that. As you can see, the traffic never even came close to the limit, and I didn't have enough monthly bandwidth to max consistent use of even 10mbps. I turned off my box, and waited a few minutes before turning it back on. The attack stopped, but it had somehow corrupted one of my SQL tables. Not a big deal... I fixed it, and everything was OK for a while. Then...

If you couldn't tell from the graph... they became more persistent. Before I go any further, I should probably explain what a DDoS attack is. This is a basic overview, and I realize it's not 100% complete or accurate.

Distributed Denial of Service Attack Explained

The Distributed Denial of Service Attack--otherwise known as a DDoS attack--is a cyber attack, where many different computers flood a target with data at the same time. The theory behind a DDoS attack, is if you can overwhelm the victim's resources while making your requests indistinguishable from legitimate request, then the host can't protect against it.

DDoS attacks are usually done through compromised machines. For example, someone may send you a virus which stays dormant until it receives a command. Upon activation, the virus will use your computer to send packets of data to a server that the controller requested. This happens to all compromised machines, which means the originating IPs of these attacks could be spread all over the world. As the victim, you can't filter out IPs or subnets at risk of filtering out your legitimate users.

At a certain point, the victim is overwhelmed with data. Either the attack overloads the bandwidth capability of the server, or it strains the hardware or software in the machine. Eventually, if the attack is successful, the server is unable to operate properly. In our case, the bandwidth capability was overloaded.

Containment Efforts

I googled this, asked friends, and even emailed professors and experts in security, asking how I could stop these DDoS attacks. I remember a lot of jargon about APF, firewalls, etc, but here's one of the more helpful responses I received:

So I had a brief chat with Dr. McHue about your problem and depending on how severity the DDoS attacks, you might be screwed. He actually recommend the IP caching filtering approach that I suggested (I was surprised... I was right) however he indicated that you will probably need some dedicated hardware for it because the lookups will eat an entire CPU, possibly more. You also want to make sure you store the IP lookup table entirely in memory. Finally, this machine has to be located far enough up the network so you minimize the packet lost due to high traffic; this might mean putting the machine at the ISP even.
Other things that he suggest is simply changing the IP of the machines you have and see how long it takes for the attackers to update. Depending on how long it take, you might get some clues as to how professional these attackers are. The second suggestion he made is to incorporate some sort of distributed architecture - multiple machines for logging in and clusters for the game state. The greater the distribution, the more difficult it is to get flooded with traffic.
What did you guys do to piss someone off enough to DDoS you?

Basically... we needed specialized hardware, or (and this is key): we're screwed. Even more importantly is the last sentence!

My... Roundabout Solution

Well, I was stumped, but I made some assertions that proved useful in solving the problem:

EternityRO is a game, so whoever is DDoSing me probably isn't really all that professional.
If I make it more expensive and risky to DDoS me, they might stop.
Additional bandwidth probably costs less than acquiring new compromised machines. Outputting too many packets from each machine would make their attacks detectable and then I could filter them out.

Now, at this point, I had consulted Razor Servers on what I should do. Unfortunately, their response to stemming a DDoS attack was to null route my IP - which is fancy for turning my machine connection off. This really pissed me off, but I never stopped using Razor Servers. Their services are generally fantastic and well priced, but their customer service needs a lot of work. Rather than keep my main box there, I rented out several virtual private servers for secondary functionality - but I'll get to that later.

I did a quick google search, found that Softlayer offered large uplink ports, a Cisco Guard, and a server in Washington DC. Having no alternative, I moved my server to Softlayer. They gave me a 1gbps--1000 mbps--line for just $10/mo! Obviously, I couldn't use that much all the time, but I also doubted DDoS attackers could keep their attacks up for all that long. I essentially called the bluff and won. After a while, we stopped receiving DDoS attacks altogether.

I guess this wasn't the best solution, but it was the solution we were looking for. I'd like to think running any project is about judgment calls - and it wouldn't have been worth our effort to find a technical solution to such a big problem. For a larger corporation, that may not be the case.

More to come... I'll post more as I write it. Follow me on twitter (@zeteg) for updates, or email me if you have questions.

36 Hours is NOT a Proper Response Time

Do you reply to text messages randomly, days after you receive them? Are you really old, and have no idea why people use text messaging? In this fully-annotated PG-13 post, I will try to explain why you need to crank up your phone etiquette. Let's start with several of my least favorite cases.

The WTF????

Usually I'm more respectful of attempts to conversation, but this is just too much. If anyone can tell me what she's trying to say, I'll buy you a drink. In case you don't know, this is an iPhone conversation where my text is in green, and my friend's text is in white. Observe:

Come on - seriously? I really didn't know how to ... SKDJGSHKDJHKJFSDKSDGN

The Delay

I end up in a lot of confused text-conversations about nothing. How can you talk about nothing? Take this wonderful example:

Thirty three hours. Thirty three.

(Kris is my English name which some friends use... because people tend to butcher Shenglong, or ask me 5 times if that's actually my real name. More on that some other time though...)

The Over-Emphasis

I love it when people text me enthusiastically; it makes me feel like I'm not wasting my time replying. Once in a while though, lack of originality and emphasis can be a little... silly:

I am tempted to model her O's as a function of messages received.

Why Text Anyway?

There has been a lot of negativity surrounding text messaging - and how carriers make a ton of money from them. I had a brief detour in youth marketing at a major carrier, and I can testify that changing send-and-end users (people who just call) was a primary objective. However, I also used to be a send-and-end user.

In the United States, calling cell phones makes perfect sense, since you have nation-wide roamaing and long distance, just as if they were local minutes. This isn't the case in Canada though, and sometimes you get charged long distance, even when dialing in the same area code. The charges are so obscured, actually, that the majority of university students in Canada don't even know in what situations they would be charged for calls.

Add on background noise, multiple conversations, and texting in class or at work, and you have plenty reasons why texting is great. The main use of texting though, is the ability to carry on passive conversations. Texts aren't as long as emails, and it allows for quicker exchanges when necessary, and slower exchanges when material runs dry.

Maintaining a good text-conversation allows you to build rapport over time, and over multiple situations. You're passively inserting yourself into someone's entire life - and any good salesperson will tell you how important that is. The reason you want to have dinner with sales targets and have golf with them, is because you want to expand the field of exposure in order to gain trust. Texting isn't as good as being there in person, but it's sure a lot better than nothing.

The 2-2-2 Rule

My obsessive use of phone and email has led to the invention of the 2-2-2 rule, which I often impose onto friends: pick up calls within 2 seconds, answer texts within 2 minutes, and respond to emails within 2 hours. There are obvious exceptions, but it's a rule of thumb to try and stick by whenever possible.

More than anything, if it's important, I'll probably call you before I text you, and text you before I email you. Needless to say, I've been told to screw off on many occasions :)

Follow me on twitter: @zeteg

Another chapter of The Tale of Eternity this week. Sorry for the delay.

Interlude: The See-Invisibility Exploit

Once in a while I will take a break from the main Tale of Eternity story, and elaborate on a very interesting issue: exploits. I will overview some common hacks, how they work, what solutions are normally implemented, and how we solved the problem. I'll try to generalize more advanced concepts so everyone can understand, but some of it might be unavoidably technical. First, there are many different ways of cheating - but in today's post, I'll just outline one.

See-Invisibility Hack

Otherwise known as the Maya Purple Hack, this is a fairly common hack exists in just about every game that incorporates invisibility. On RO--and probably most other games as well--this hack works because of unnecessary information being sent from the server to the client. There are semi-logical reasons for it, but in the end we have to blame it on poor programming.

The Reason

Normally in a visible-player setting, the client needs to tell the server "Hey, I'm at square x,y", and then the server checks for all players within the area (x +/- 14, y +/- 14). For each player it finds, it tells them, "Hey, this player is at square x,y", at which point each recipient's client translates that information and shows the player visually. Pretty simple, and it makes sense.

Of course, location data isn't the only data that's sent. At the same time, sprite data (how the player looks), class data (what class the character is), and a bunch of other data is also sent--including visibility data. With the way the RO client is programmed, received information that isn't recognized by the client results in an error dump--a message with lots of Korean code, followed by a client crash.

I'm guessing it's because of this, that some genius at Gravity Inc. decided to implement invisibility as it works now. With invisible characters, the same aforementioned process happens, with one key exception. After the client receives the location data for the invisible player, it also receives visibility data, informing the client that said player is invisible. In turn, the code tells the client to show nothing, rather than a player. Okay, this solution works as long as no one tries to tamper with the game client.

The Problem

There are two generic ways to exploit this: hex editing, and packet filtering. There is a third way, but it's game-specific and not worth discussing at length.

Hex editing works with the hexadecimal makeup of a program. Just like a program can be represented in binary as a string of 0 and 1, it can be represented in hex. With unencrypted files (such as the RO client), hex editing is extremely easy, and values can be changed in a heartbeat. Unfortunately, such was the case for the RO client. For clarification, imagine the following piece of pseudo code:

if (visibility = no), then {player = not visible}

Essentially, what you're doing when you're hex editing is changing it to:

if (visibility = no), then {player = visible}

Hey, now you can see invisible players. Granted, it's a little more complicated than I've described.

The second method is packet filtering. By ignoring visibility packets and filtering for position packets, a user is able to translate the packet data directly in order to see hidden players. This can be solved with a good encryption technique, but encrypting every packet for real-time gaming proves to be a real problem. Some problems originate from increased server load, and others from mirroring the decryption on the client.

If you use a basic encryption technique, it's going to get cracked; that's a fact you can't avoid. In order to encrypt and decrypt packets on the server, you need related functionality on the client. This ultimately means you're delivering your encryption system to the exploiter. He's going to diff your files, collect packets and analyze them for patterns, and eventually try to solve your encryption. Sadly, the more secure your encryption technique is, the more load it imposes on the server.

Let me give you an example of server load: On RO, potions can be consumed at a most, 10 per second. With 2,000 players all consuming on average 5 potions per second in Siege War, there are about 20,000 MySQL inserts per second, counting inventory and logs - which is taxing enough by itself. Try to blowfish each of these 10,000 packets, and a nightmare ensues.

The Solution

The traditional methodology in addressing this problem strikes an uncanny resemblance to struggling competing products: a functionality war. The builders keep improving on their old techniques marginally, and after a week or two, the hackers catch up marginally. The cycle just keeps continuing, eating up a ton of time, and not really getting anywhere. You see, when you introduce a technical problem, the exploiters--who are people who do this for fun, just because they can--are excited about the challenge.

To disengage people from breaking the encryption, you can either introduce a revolutionary technology so profound, that it's no longer fun to solve, or find an alternative method that can't be attacked directly. A technical solution was never finalized for the invisibility hack, and there came a point where it became a big problem on Eternity.

Because we had no reliable way of catching and proving cheating, the bad players started using it - giving them an advantage over people who didn't use it. The good players, feeling the situation was unjust, decided to level the playing field and use it too. Soon, a large portion of the server was using this exploit, and we were receiving a ton of complaints.

Then, genius: Somewhere in the middle of version 2 (I haven't gotten that far yet in my story), I was discussing this issue with my co-admin, Griffin. I rebooted my brain, and proposed an idea for an alternative way of catching these cheaters - we'll codename it Project Stalker to be hip and unique. Project Stalker involved several steps of logic.

There are only a few ways in the game to reveal a hidden character.
By manipulating game mechanics, it is possible to make a "hacked character" undetectable by normal means.
Hence, if anyone sees this character, they must be cheating. But how do we tell?
When a player hovers their mouse over a character, a packet is sent to the server, requesting the name of the selected player.
Thus, if we receive a name request packet on an undetectable character, the requester must be cheating.

Griffin understood immediately, and proceeded to code it into the server. He had the server spit back logs of players who try to request a specific character we hard coded in to be our undetectable character.

Next Siege War, we tested the system. Griffin ran around as Project Stalker, and the server spit back the names and IPs of every player who requested his packet - all the while without letting them know what was going on. The numbers were shocking. Over 50% of the players were cheating! Instead of banning everyone, we alerted the guild masters of cheating players (some of which were cheating themselves) and made a general announcement: "We have a way of catching you. We're letting today slide, but next time, you will be banned."

We succeeded. People were scared, and they realized we weren't lying. The amount of invisibility-exploiters dropped from over 50% to less than 1%, and we went through several Sieges without catching a single cheater. This remarkable feat gave us a lot of credibility in terms of catching cheaters.

More to come... follow me on Twitter (@zeteg) for updates

Why ISPs Shouldn't Ban MAC Addresses

I'm very fed up with the University of Toronto, and its terrible network management. Last Saturday, I was cut off from the internet at about 11:00 PM for no apparent reason. I hadn't touched any cables (I'm hooked up by a cord), and the other people in this suite had perfectly working internet. I sighed heavily, scooped some ice cream, and proceeded to diagnose the problem.

The little connectivity symbol on my startbar was acting strange. Once in a while it would appear normal:

But sometimes, a yellow exclaimation mark would appear, signaling that something was wrong. After running a diagnostic test, Windows 7 told me my configurations were correct, but it couldn't connect to the primary DNS. I tried disabling and re-enabling, jiggling my cord, and even ran a bunch of ipconfig commands in hope it would be fixed. No luck. Here's what it would look like:

Ok - I had three ideas:

My ethernet card was broken
My ethernet cord or socket was really broken
My MAC address been banned off the network without notice

Let me rewind. I'm currently living at the University of Toronto for the summer for an internship - I don't actually go here. My $600/month under a 4 month contract is supposed to include internet service, and I have been very careful not to violate any of their sensible rules. I say sensible, because I'm pretty sure a failing law student drew up their contracts. I point you to one of many issues with the Occupancy Agreement, for example:

14. The Resident will not keep any firearm, fireworks, weapon, explosive, animal, fish, reptile, insect, bird in the Room, Suite or Residence

Read it carefully - multiple problems should jump out at you. This is a completely insensible request, and every suite in the entire building is in violation. So once again, I obey all the sensible requests.

To test whether my ethernet card was broken, I tried a wifi connection, and also connected my computer to my friend's computer through the same ethernet cord. I then connected my computer to my friend's wall socket. This tested both #1 and #2, and showed me that for sure, I had been banned.

But why? I didn't get a knock on the door, a message, or even an email. Okay, network administrators are sensible people - I'm going to go talk to them. If they're up banning me at 11:00 at night, they should probably be awake right? Wrong. My suite mate told me that apparently (I can't confirm this) this university has software that flags individuals. Then, apparently there is a delay between flagging and banning. This means that once you've been flagged, you will be banned at some random time in the future. I really hope this is not true... but evidence thusfar corroborates the theory.

I went down stairs and I was told that the technicians "do not have a set schedule". Furthermore, I was told I could not contact them directly, and that I could fill out a form and they would find me "when they have time." However, being a long weekend, that might not be until Tuesday. I was about to have a verbal scuffle at this point with the desk staff, but realized they couldn't do anything about poor policy. I asked for a contact number, made them aware of my situation, and went up stairs to sleep.

The next day, a technician comes to my door and tells me he can't do anything to help me. Yes, he came to my door to tell me this. He further assured me that he would find out what was going on, and at least let me know by Tuesday. He couldn't even find out why I had been banned, and blamed it on the "central network" - and his tone suggested it was some sort of omnipotent power. He further suggested that I may have been banned in error, and that I should find alternative ways to access the internet.

Good point, technician! I might have been banned in error. I went back to my computer, and decided to just get myself online, because it's not like they intended to ban me - right?

Good thing Microsoft smartened up and made it easy to spoof your MAC Address in Windows 7. I didn't even have to do any tinkering. So there it was, I changed my MAC Address and what do you know? This happens:

This is the standard page, telling you to authenticate yourself. I just used the same information I signed up with originally, and it worked. Had they banned my records, could easily have wrote a script to brute force every room-birthday combination. Of course I didn't, because I wouldn't want to do anything bad. They hadn't banned my room, so it must have been a mistake. I ran a security check, and bamn:

I was back online! I could hang out with all your wonderful people again:

Subsequently, I have been banned again yesterday, and just now again about 50 minutes ago. The funny thing is, I figured maybe I had broken one of their rules.. so I did a test. Between when I was first banned and now, the only things I have done are:

Visit HN and read articles
Use Wikipedia
Check my Gmail
Use Facebook / Twitter / LAL / G+ / MSN / Skype / GTalk
Google random things here and there

No video streaming, no downloads, not even YouTube. It's almost Thursday, and no one has gotten back to me yet. I know this isn't a big technical accomplishment, but it's a great illustration of outdated techniques and poor implementation/service. At the very least, it's a semi-funny story to read (I hope).

The Tale of Eternity: Part 2

In This Series...

If you haven't read the prelude or first part, I strongly suggest you do before you read this segment, as this might not make sense otherwise.

Life Beyond Death

I made a lot of enemies before Eternity even launched. As it turned out, our early adopters didn't just leave their previous servers - they took it upon themselves to hasten the eventual demise. With nothing left to lose, they marched in proud defiance of rules, conducting all sorts of unsolicted behavior. Advertising for Eternity, obvious botting/packet-sending, and DDoS attacks were just a small part of the mess.

I imagined most marketing managers would be ecstatic to have their customers attack their competitors unguided - It would be some sort of pre-emptive victory. I certainly felt this way, and aside from the unconvincing PR speech about how "we shouldn't cause damage on other servers - we're better than that," I did very little to stop it.

Looking back, it's blatantly obvious that my long line of mistakes in this whole ordeal started here. I'm not going to spew moral bullshit and tell you how I should've abandoned my sanctimonious stance. Regardless of whether I erred morally, it's a failure in foresight that haunts me most.

I should have pulled a Gandalf and voiced an epic proclamation that would have echoed for generations to come. "You shall not pass!" - or at least, "We will punish you here for bad behavior on other servers!" is what I should have said. Yet, I didn't. I chuckled at their misfortune, eyes gleaming and mouth watering in anticipation of the power and glory that lay ahead.

You see, back then I believed morality played no role in business - which is not what my Business Ethics class taught me. Though to be fair, I'm not really sure if that class taught anyone anything. Ethics classes tend to ramble on about what some 200-year old skeleton believed - and not once has anyone given me a convincing reason that ethical choices should be practiced in business when these choices are in opposition of obvious benefit (not just immediate profit).

If you're in the same boat as I was, I'll share with you the two most convincing reasons I learned (It's greyed out because it's kind of boring):

Modern human society is based on a sense of trust. We trust our government (or used to) to enforce the law, we trust our neighbors not to steal our things, and we trust teachers we barely know to take care of our children. It makes sense, that a sense of right and wrong--the fundamental basis of morality--is derived from our survival. One could infer, given this line of logic, that when involving multiple parties in a public setting, morality is also the superior choice. Yes, this argument has logical deficiencies, but this isn't a debate, and I did not describe what I'm trying to say perfectly. Hopefully you'll extrapolate and derive some value.
Even if your business doesn't prosper, making an unexpected moral choice leaves a profound impact on those around you. Unfortunately, this only really works if it's a heavily publicized decision. I read an article today detailing Benjamin Franklin's refusal to have the Stove patented, because he believed others should have access to it as well. Wow, makes him seem like a good guy! I think it's natural for people to respect genuinely good actions, whether it's something they would've done or not, and having this sort of respect can make you life far easier in the future.

I talk about a lack of foresight, but the truth is, throughout this project, I've made some extraordinary insightful decisions. I've received praise from players, staff, and even previous competitors. Many people say praise is a dangerous beast that consumes its target - but I rarely see a modest man swell with pride. It appears that praise only breeds haughtiness in cases where the praised are already egoistical.

That was me. I did not believe that anyone could make better decisions than I could. I thought I was the smartest guy around, and I could easily account for any amount of steps my competitors were planning. I was sure I could predict the market - and you can't imagine the depth of my smile when a player said, "you're an oracle... you just see everything."

The disaster that my amoral actions wrought were only revealed half a year later when players started to do to Eternity, what they had done to our competitors (I'll get to this in more detail when I get to that part). If I had set precedence for how all servers would handle bad behavior back when we had leverage, Eternity version 1 would have lasted much longer. In fact, it could've been my chance to establish and lead a system akin to patent laws in the United States.

But no. I told myself I could predict player actions, and went on to enforce a set of rules that actually did work... until players adapted. Your users will always adapt if they force any sense of irrationality on them, even if it makes sense on your end. Much like viruses and bacteria, an adapted version of the original can be more harmful than the originating strain. Now armed with experience, I heavily advice against forcing adaptation on your users. I see Apple doing this, and I feel quaintly nostalgic. Much like Apple, I should have found an alternative strategy... kind of like how Diablo 3 plans to deal with gold farmers by introducing a new business model aspect.

What's a Server?

Just so you don't think I'm understating how much I knew - I spent a good deal of time trying to figure out the difference between shared hosting, virtual private servers (VPS), and dedicated servers. I guess some part of me always imagined that RO ran like a regular website. Permissions? Bah! Memory? Who needs that!? Wait wait... what's memory again?

By this time, I was googling things left and right, and trying to find a cheap dedicated server. After browsing around, I eventually ended up with Razor Servers both because they were cheap, and because their servers were based in DC at the time. You see, DC was the perfect spot for a gaming server, and Razor Servers in particular offered very low latency to most areas we were serving. Eastern Canada/USA had 30ms ping, west coast had <100, and Europe had less than 140 - which is much better than people were used to experiencing.

RO relies on reaction speed, and an extra 30ms of ping is often the difference between winning and losing a fight. Latency and reliability were important, and all the marketing materials on their website told me they provided both. Yeah, I realized this wasn't all true, but I also realized that because they claimed it, I could tell my players, who in turn believed me since a server provider can't possibly be exaggerating.

But heck, a dedicated server was going to cost me a lot of money each month.

That's $100 I didn't have. Gosh - time to think outside the box again: time to go to sleep. You see, half of my thinking is done on the border of consciousness and sleep - sometimes enter a blissful state where I can think of thousands of concepts in a split second and analyze them too. I don't know if this is real or just imagined, but regardless, most of my ideas come from either when I'm falling asleep or I'm waking up.

Never before done in the RO world - I asked for pledges. Before I explain the details of this financial instrument (I made more complicated ones later on), let me go over how finances typically work for any given private server:

The Problem with Old Cash Shops

Every server needs money to run, and very few server owners bootstrap their servers. Usually they make an initial investment and either go for a break-even model (rarely by choice) or profit from donations. They're called donations, but they're not really donations. They're payments, in exchange for some type of virtual currency or service. I guess Farmville has made this concept pretty popular, but when we did it, the concept of Cash Shops had just sprung up.

Most servers offered substantial rewards for donating, and gave away powerful items that otherwise couldn't be obtained. Pause and think - what could be wrong with this model? Well, the people with money usually aren't the hardcore players. To put it in more offensive terms: whales are never athletes. Because of this, every server using this method eventually developed a secondary market for the resale of donation items.

There were two forces that eventually drove down the numerical worth donation items even in a heavily inflating market: progression and liquidity. Players will always progress faster than anyone can reasonably introduce new donation items which were at least somewhat balanced. To add to this, once donation points are spent, the resulting item loses a significiant portion of its value because it is now restricted to certain classes and builds (cut in demand). The whales never colluded, which further allowed the hardcore gamers dictate the digital worth of each dollar. This became a pivotal strategy concept for Eternity.

So, by now you're curious. How much money can an RO server make? I'm not going to reveal Eternity's financials (although I will mention them later on), but a server with 2,000 players peak and statted donation items can make upwards of $20,000 a month. Not just a "game" anymore, now is it?

Eternity's Decision

I told everyone we would not have any statted donation items - knowing it would significantly cut potential profit. However, it aligned with our culture, and it served as the first step to the cash shop Nash Equilibrium. Yes, this is one of the reasons we had so many potential players. We screwed over everyone including ourselves in terms of $value/player, but increased our number of players significantly. Worth it? We'll see.

Instead of items that add power, we introduced items that would add aesthetics--pretty hats.. useful and pretty hats. I came up with the idea of synthesis, and you can read my poorly written code here. It's just a NPC script, and anyone with any experience in programming should understand basically what it does. I've never given out a script before (ever), so feel special!

Synthesis was our selling point, and it's an idea that's been used (I don't dare say copied because I have no proof I was even the first to think of it) by various other games that now exist. Essentially, a player could merge a good looking item with the stats and abilities of another item.

Complex Financial Instruments

I went through a phase where I really wanted to be an investment banker. Why? Well, one night, a friend and I were lying in bed chatting (platonic), and I brought up how I don't usually wear formal shoes to interviews even when I wear a suit. She asked me why, and I told her it's because I value performance over appearance, and that formal boots are not really good for anything practical. She hmph'd and declared "Goldman Sachs would never hire you!" I responded, "What's Goldman Sachs?" and from then on, I was hooked. A challenge had been issued, and I was determined to join Goldman Sachs.

In those days, I had the pleasure of reading about the IB industry, and about financial instruments. I think it's from there, that I got my idea for pledges, and later in v2, options. I asked the population for an early donation, on the promise that if we launch, they would be rewarded double. It worked. We fund raised a good $1,000 in the first two days, and I was in financial bliss.

Rush for Substance

Oh boy - now we had more willing customers than we hoped, a ton of money, but no product! This was like the reverse situation of a usual startup, and it sounds better than it really was. As I have mentioned, I had no idea how to set up a RO server, much less how to "fix lag" and make other promised things come true. Ayumi? Autumn? Prodigy? They had no idea either!

Oh right - by this time, we had added a fourth co-founder at Ayumi's request. It was actually the guy she was e-dating. I'm not sure why I willingly agreed to dilute my interest, but it probably had to do with a fear of incessant whining. Regardless, it was done, and he doesn't really play much of a role in this story. The fact remained - no one knew how to do anything.

Enter Articulus - savior, benefactor, and all-around awesome person. Articulus graduated engineering in California, and founded the server I had played on before, so it was only natural to approach him for help. Rather than setting everything up for me, he decided I had to learn to set things up myself... starting from the basics: CentOS. Why CentOS, I still don't quite understand. I struggled and struggled, but eventually got the OS and everything set up to run RO. I set up subversion on the system, and configured the proper settings with a lot of help from him. He's the kind of developer that every company should be looking for. It's genuinely difficult to find someone smart without a massive ego, who goes out of his way to help others with time-consuming problems.

I know setting up Linux and configuration doesn't sound like a lot, but for someone who had no idea what he was doing, and still going through school, it was tiring. I had a few sleepless nights in preparation for launch, and it's safe to say my nightly dreams transformed into a dedicated server committed to configuration. QA and testing became second nature, and I I realized I had a penchant for finding esoteric bugs.

The Payoff

It was launch day, and everything had been set up and tested.

3... I turned on the login server

2... I turned on the character server

1... AWW SO CUTE

Yes, if you ever have the luxury of building connective software that requires multiple parts to function together, please watch how your eager customers spam your login. I swear some people wrote bots just to log in. I had 200 people spamming the login every 5 seconds. It's interesting to note, that we later used this fact to find bots.

After watching with a blissful smile, I remembered I still had to launch map.

0... Map launched!

I watched the character count on the server skyrocket to 350 in the first 2 minutes. Ten minutes later, user count was at 500, and peaked out at about 750 two days later. That's right - 750 people online at once. Our team rejoiced, and we honestly thought the hardest part was behind us. Boy, were we wrong.

More to come... I'll post more as I write it. Follow me on twitter (@zeteg) for updates, or email me if you have questions.

The Tale of Eternity: Part 1

In This Series...

The First Investment

We were an internet company rendering an internet service - so it was only fitting that we should have a website. Sadly, I was a money-starved student, and didn't have enough confidence at the time to invest any money into a domain and hosting. Luck should have it, I was an entrepreneur with an idea, and like every entrepreneur who needs money, I asked someone else for it. Humiliatingly, I don't actually remember his name, but this kind/evil person gave us web space, and bought eternityro.net for us. We were friends - with emphasis, because I didn't really enjoy his presence at all. He wasn't very cooperative, and he was extreme emotional and belligerent. Actually, overly-emotional people seem to be a theme in this series, and if you take one piece of startup advice from me, it should be this: don't place any sort of power or leverage with any emotional people.

But ultimately, he was willing to fund us when nobody else was, and that made him an asset at the time. I emoted smiles to the outrageous things he'd say, and softly acquiesced to all the ridiculous things he wanted in return. He didn't want to be paid back (that would've been acceptable) - he wanted us to cheat for him, which was the very corruption we started Eternity to eliminate. Funny how things work out. I took the funding even under these conditions, because I knew I'd weasel my way out of it later. Facing this decision again, I would've found a smarter solution.

Website, Forums, and IRC

It was done. We had hosting so I started working on a website. I remembered my friend Jason (the genius who made MangaMasters and the CMS behind MapleTip)--a wonderful hacker--had helped me make a webpage for a board game that some friends and I wanted to release when we were younger. Here's a picture of the Dash Gaming game. We had the projections, company culture and strategy all fleshed out, which was a pretty big deal for us, at our age. Sadly, the product never launched as real life got in the way.

Anyway, I adjusted the site he kindly made us into the prototype EternityRO page and used what little php I knew from trying to write my own MUD to make it seem a little more sophisticated. It's probably full of security holes, but back then, everyone was rooting for our success so it hardly mattered. We were the new cute-girl at school. Everyone either wanted us, or wanted to be like us. Okay, bad analogy - excuse me.

Next, came the forums. The main forum softwares available back then were Invision Power (IPB), vBulletin, and SMF. Yet, we had little choice, considering every server our target population has ever played on has used IPB, and I knew how much people resented change. Problem! The licensing fee for IPB was $150 - which was $150 we didn't have. Ayumi came to the rescue, and told me she found a copy (license) for IPB. I realized in the back of my head that this wasn't a legitimate copy, but at this point I just wanted to make Eternity a success. "This is legal, right? *cough*" I asked her. "Yes," she replied. Good enough for me.*

* I later corrected for this mistake, by purchasing an actual license for IPB in addition to their support licenses for the next few years.

Next was our support and discussion channel. Not every game needs this, and there are drawbacks, but IRC-type support can also be valuable for a number of reasons:

Players can give other players help without using your time, even when you're sleeping
You avoid negative reviews by providing an instant channel for support
You can receive immediate and detailed feedback about what you're doing wrong
You can create a core community, and bind people to your product based on their interconnectedness, rather than loyalty

We settled on a free IRC channel on the deltaanime.net server - which honestly, is a terrible server. But, that's what everyone else was using, so we went along with the flow. DeltaAnime, otherwise known as DA, actually ended up hosting eAthena for a very long time, and branched their own VPS business that way. There'll be more about this later, when I describe my abuse of advertising channels.

The Landscape

Before I go any further, I need to retrospectively* delineate the market landscape we were about to enter, since without a basic grasp of the situation, what I'm about to describe won't make much sense.

The RO private server market was an unmitigatable disaster, and the only factor driving entry was irrationality. If anyone had paused to think about their chances of success, or even their projected lifespan if successful, no one would have started new servers. Yet, they did - in droves. Even in present day--now that the market has died down significantly--if you google "top ragnarok servers", you'll find a site ranking at least 500 of them. You'll also find other sites that list even more.

Now that we know there are hundreds, if not thousands, of private servers, you'll have to take my word that new servers have an average lifespan of 3 months. It takes the average hopeful about one month to realize his dreams aren't going anywhere, and another two months to admit it. These are qualitative conclusions, but considering my expertise in the area, I feel qualified to make these as if they were based on quantitative facts. Why does this happen? Every server owner knows that he needs a large population to sustain the server, because that's what all private-server players have come to expect. Yet, they only focus on trying to get more population, because they believe their servers are perfect. I don't disagree - they very well might be. Some of these unknown servers developed features that my entire development team couldn't. Yet, they're dead.

Small tangent: I sense a lot of hostility in the HN community, from developers towards business people. I think, for the most part, this is generally justified, since most self-proclaimed business people are people who don't have any other skills... and "business" is generic enough to fake. I really don't know where or how you learn business skills, but I feel it comes naturally to some people. In this situation at least, it wasn't our development team--as awesome as they were--that allowed us to thrive.

To summarize: this market sucks. Did it affect us? Yes, it worked to our advantage, but also limited our size.

* -Shiver-...In my current line of work, everyone absolutely hates the word "retro".

The Chicken and the Egg

This was a MMORPG (Massive Multiplayer Online Role Playing Game, often confused with Many Men Online Role Playing Girls), and the key performance indicator that everyone cared about was population. As aforementioned, players had developed a cynicism towards servers, and only wanted to play on ones with a large population (and therefore didn't wipe). If a server started out strong, late adopters would flock in as well, bolstering the population. If a server started weak, most of the early adopters would be gone in a day or two. This led to a lot of spam on part of server staff, trying desperately to get players to go to their server. "We have better features! Come play here!" they yelled, and no one listened.

I approached things differently. I talked about vision, and instead of telling people what we had, I told them what they wanted, and why they wanted it. I told them what they hated, and why they hated it. If you want to speak in business terms, I aligned our customers with our corporate culture. Once I elucidated their own desires, I connected our server as the answer. I actually just read Steve Yegge's response to HN commenting about suggestion, and it's fascinatingly related: I kept telling people we would be the final answer, and people believed it, even though I didn't offer any proof. In fact, at that time, I didn't even know that eAthena ran on Linux, or how to compile code. There was so much blind faith, that believers championed my ideals, and preached to their friends like it was their own idea. These people actually ended up being the most valuable part of my community.

It was time to spread the word that EternityRO existed. What advertising channels did we have that were free? Hmm, let's see... we could... ... ... spam other servers. I mulled this over for a little while, and came up with a tactic I called tactical infiltration - which at best, falls into a legal and moral grey area. Instead of spamming directly, my idea was to go on another server's forums, and create a topic called "Quitting, hosting a contest", whereby a non-existent character would pretend to have a lot of gear to give away. He would then post some text akin to "I'm quitting this sever for EternityRO" without leaving a link, as to avoid suspicion. The contest would list items, and say "the 100th person to post the name of the item they want gets it". Commence flooding. I did use this technique later on, and it worked amazingly well. However, I didn't need to do anything of the sort for our first launch.

Ayumi told some of her friends, who talked to me on IRC. Within the next three days, our IRC channel had grown from 3 users to 25, with most of these people being either socialites in the community, or guild leaders. I boasted, I smiled, and I made it seem like I was listening to what they had to say. They asked me for a release date, but I told them the release would not be for another 2 months. Most release dates are within a week of announcement, but Eternity would be in two months time.

My reasoning was simple. When we released didn't matter, since we were competing as a new server, against large late-stage servers. The longer we wait, the more upset their players become, and the more leverage we have. The longer we wait, the more time news of Eternity has to spread, and the longer people have to consider it. The epiphany was: by that time we had become an established thought, and not something that can just be forgotten.

You see, servers are inherently unstable for players, and they need to be on constant watch for the next server that everyone else might flock to. Everyone else seems to have forgotten that large groups don't make decisions as a group - they make the decision they think everyone else will make. Therefore, it's only necessary to make everyone think everyone else might go to Eternity when it starts. If they believe there's a chance, they'll make an account and start playing on the first day to establish an advantage for themselves and their guild. Once they see that everyone else is doing the same thing, it's only natural to assume everyone else has made their decision. It's a little funny. Instead of having sheep being led, we have sheep leading sheep.

That was my plan anyway - but plans rarely work out the way you want them to. I forgot to account for demoralization in my favor. After hype about EternityRO picked up, people just stopped playing on other servers completely. They felt that any more time spent on those servers would be a waste of time, since Eternity had so clearly become the better choice. I asked them why they thought this, and they echoed to me the same reasons I had given them weeks ago. The stage was set, and a third of the English-speaking low-rate community was ready to move to Eternity on the first day.

(more to come in future posts... I'll tweet them as I write them! @zeteg)

A Prelude to Eternity

In This Series...

It's been more than three years since I started the EternityRO project, and there isn't a single person other than myself, who knows the whole story. Originally, I had planned on letting the story die, but after reading many stories on Hacker News, I decided I should do my part for the community - as little as it is.

Here, I'll share how I started a private game server that grew larger than official servers - almost 300,000 registered accounts and almost 4000 simultaneous players. I'll outline how I diversified my hosting, dealt with players from over 107 different countries, escaped a DDoS attack, created alternative ways to catch hackers, and ended up keeping one of every 20 people I hired.

Although it wasn't really a startup, it really felt like one; I really hope this story is one that actual founders can relate to... at least those of you who jumped into the fire, completely unprepared. Unfortunately, before I get to the meat of the story, I need to brush over some quick facts. I'm linking everything (except to official servers) for easier understanding. If you want to get into the meat for the story, skip to the next post.

So what is EternityRO, and why was I able to run it?

EternityRO was a private Ragnarok Online server, made possible by an emulator. The open source group, eAthena publishes a set of emulator code based on the original Aegis software that Ragnarok Online's founding company, Gravity Inc, runs. Thanks to17 USC 102(b), some privacy laws, and inter USA-Canada-China-Korea legal confusions, running emulated software is apparently legal, as long as you make it really annoying and costly to find you. Honestly, I couldn't even guess what country's jurisdiction I would fall in anymore. Thinking back, I probably should have gotten a lawyer's opinion. If any lawyers are reading this and want to send me a note, I'd love to hear it.

So why would anyone play a private server hosted by a kid (well, young adult)? Simply put it, official servers have several problems:

Official servers charge a monthly subscription fee
Progression is often slower on official servers (1x rates)
For some reason I just can't figure out, official servers tend to lag
Bots run rampant on officials, as part of their marketing strategy is to allow trial accounts.
Cheating (WPE, OpenKore) run unchecked, because HackShield is garbage.
Inter-Guild competition is more intense on private servers*
Private servers are more agile, and quicker to balance issues*
Private server staff speak English... and other languages. Plus we have no life, and are available 24/7**

* Not true before EternityRO. Eternity set the stage, and ingrained these concepts into the community. For the RO community at least, #6 has become the leading force, driving almost every server hop (change of servers).

** Sarcasm... but was often the expectation

My Background

I started Eternity at the pleading of two co-founders at the time, screen names Ayumi and Autumn. By this time, I had been playing the game on a different private server, and had gained some knowledge of the game's mechanics and scripts. In this group, I was meant to be the developer - although really, I wasn't qualified. Ayumi, the real pioneer of this whole project was an emotional socialite. Possessing e-bewbz in a predominately male community grants you special abilities, otherwise not available. Plus, she was e-dating one of the celebrity players (he made lots of movies) at the time. Autumn--one of Ayumi's friends--was brought on board as a co-founder to head design. In retrospect, I'm pretty sure Autumn was just Ayumi's way of seizing majority control in a voting situation. Sneaky.

So, as a full time student, I agreed to start a new RO server with two other full time students who I had never met. Without any concept for how much time this would eat up, we started dreaming the founder's dream. We were the wizened oracles, armed with knowledge of all that was wrong, and the passion to set it right. And really, that must be how it all starts, because no logical-thinking person would logically jump into their first startup with full time commitments on the side. There's one big difference though: when you set out on a startup journey, usually at least one of your co-founders is a decent coder. I was the developer here, and all I really knew was some basic HTML. Needless to say, probably not something YC would have funded!

Almost There...

I didn't really know why I was chosen to be a co-founder. I had almost no marketable skills, other than up talking every mound of potential into a mountain of accomplishment... otherwise known as boasting. My mother, one of the most accomplished and deserving academics I know, would like to take some time at this point to remind me that cow blowing (boasting in Chinese) is actually a very essential skill in any line of work - and academics in particular. Over the course of these few years, I've realized that marketing and boasting are really just two words for the same thing, and it's become one of my most valuable assets.

So... without any plan, I started boasting. To everyone.

The next part of the series is here:

The Tale of Eternity: Part 1

Shenglong’s Blog

A blog about technology, life tricks, and my adventures

The Tale of Eternity: Part 3

36 Hours is NOT a Proper Response Time

Interlude: The See-Invisibility Exploit

Why ISPs Shouldn't Ban MAC Addresses

The Tale of Eternity: Part 2

The Tale of Eternity: Part 1

A Prelude to Eternity