Overall goals
- A gaming client to render the game for the user.
- Gaming servers to interact with the gaming client.
- A Web application server to integrate with the gaming servers and clients.
- A database server to persist and retrieve data.
2010年10月17日 星期日
Building a simple yet powerful MMO game architecture, Part 1: Introduction
From: http://www.ibm.com/developerworks/library/ar-powerup1/index.html
Summary: There is growing excitement among business, education, and government institutions in massive multiplayer online (MMO) virtual-world games and how they can be applied to business and educational needs. MMO games offer tantalizing new ways to learn, entertain, collaborate, socialize, visualize information, and do business. In this series, learn about an architecture based upon the first 3D MMO game from IBM, PowerUp. This first article will begin to show you how to build a flexible and powerful MMO game architecture that is quick and easy to implement.
Introduction
Massive multiplayer online (MMO) games are Internet-based video games that can accommodate hundreds—or even thousands—of concurrent users. A defining characteristic of most multiplayer online games is that they present a single, integrated, persistent gaming world. World of Warcraft and Second Life are examples of MMO games.
The Halo and Counter-Strike series of games are examples of basic multiplayer online games (MOGs). Even an online game of Yahoo! chess can be considered an MOG. However, these examples are not MMO games because they lack an integrated, persistent gaming world. MOGs typically consist of a list of separate, nonintegrated, multiplayer matches, and the matches continuously recycle through a series of rounds. For example, in Yahoo! chess there is no persistence: the game ends and starts anew each time the match ends. And of course, there is no semblance of a single, integrated, chess game "world."
There are several MMO game genres. Massive multiplayer online role-playing games (MMORPGs) are the most popular.
Part 2 will explore technical details of the architecture, including the functions and calls for integrating game clients and servers with back-end systems.
Part 3 will assess whether the architecture addresses the overarching goals (as shown here in Table 1), reviews the capabilities and limitations of the architecture, and includes lessons learned.
Others examples include massive multiplayer first-person shooter games, MMO racing games, and MMO real-time strategy.
In this series, learn about an MMO game architecture that accommodates almost any MMO game genre. The architecture underlies several IBM projects, including PowerUp, The first 3D MMO game from IBM available to the public.
This first article outlines the architecture, specifications, and intended functions of the architecture. You'll learn about the high-level and detailed architecture design.
Figure 1. IBM MMO games (PowerUp at left)
--------------------------------------------------------------------------------
Overall goals
An MMO game architecture should address the specifications and goals in Table 1.
Table 1. Specifications and goals
For... The architecture should...
Scalability Allow for any potential number of concurrent users. It should also scale with relative ease.
Flexibility in deployment Allow for a wide range of gaming server configurations and updates. Ideally, the server configurations should be relatively easy to reconfigure. Updates should be transparent to the end user.
Flexibility in gaming design Minimize constraints on game designers, and facilitate the ability to design an expansive, integrated gaming world.
Performance Perform smoothly and quickly. Provide performance-related functions, such as game-server load balancing and monitoring.
Functions Provide a means to perform functions for an MMO game. A feature might be the ability to persist and retrieve game data, such as player and game-related data.
Security Be secure and not expose any security risks.
High-level architecture
The MMO game architecture described in this article consists of four major components:
Any Web application server (such as IBM WebSphere®, BEA WebLogic, or Apache Tomcat) will suffice for the architecture described in this article. Any database server will also suffice (IBM DB2®, Oracle, MySQL).
This article focuses on the integration between the gaming server and client code with the Web and DB2 back end, and not on the gaming client and server or the back end themselves. It doesn't address how to create a building in Torque Game Engine Advanced (TGEA) or what networking protocol TGEA uses for the gaming client-and-server networking. There is ample documentation on TGEA at GarageGames. The architecture in this article is somewhat predicated on the Torque Game Engine Advanced (TGEA) from GarageGames. However, other multiplayer gaming engines provide similar networking functions to TGEA, such as:
The integration between the TGEA client and server to the Web and database back end described in this article is performed with HTTP requests. Other gaming engines also provide this function.
All multiplayer-capable gaming engines provide gaming client and server networking, although each gaming engine implements the actual gaming client-and-server networking in its own manner. Therefore, there will be some idiosyncrasies with the TGEA gaming client-and-server networking code that cannot be applied to other gaming engines.
Figure 2 shows the high-level MMO game architecture. Several IBM MMO games use this architecture.
Figure 2. High-level MMO game architecture
The architecture consists of four major components: the game client, game server, Web application server, and database.
TGEA client (game client)
The client machine 1...N represents the user's machine running the TGEA client. 1...N indicates that there are more than one concurrent machines (users).
With the HTTP client, TGEA can send GET requests to a Web server and process the returning page sent by the Web server.
The Torque Game Networking Client is the default game-networking client code that allows the game client to talk to the TGEA game server.
TGEA server 1...N (game server)
1...N indicates that multiple TGEA servers can be run on a single physical server. Multiple TGEA servers can also be run on a particular processor within a server. By default, TGEA allows for 10 instances of a TGEA server per processor.
Physical server 1...N is the actual physical server that runs the TGEA server. 1...N indicates the architecture accounts for multiple physical servers running TGEA servers.
Torque Game Networking Server is the default game-networking server code that allows the TGEA servers to talk to the TGEA clients.
With the HTTP client, TGEA can send GET requests to a server, so this indicates that the TGEA server can also make these HTTP requests.
Web application server
Physical server 1...N are the physical servers that run the Web application server.
HTTP server is the Web server that processes the HTTP requests from the TGEA client and server.
Database connection is the application programming interface (API) that connect the Web application server to the database, such as Open Database Connectivity (ODBC) or Java™ Database Connectivity (JDBC).
Database management system
The database server to persist game data.
The design is a standard three-tiered server architecture, where the client, the Web business-logic server, and database components are modularized. The benefits of a modularized three-tiered model are well established. A modularized design provides standardized interfaces and makes it easy to update any particular module. The gaming servers can also be considered clients of the Web server.
The TGEA client and TGEA server are a standard MOG that TGEA (and most other multiplayer gaming engines) provides out of the box. The Web application server and database management server provide the functions to implement an MMO game, or a persistent, integrated gaming world.
The Web server serves up Web services that can retrieve and persist data. This loose, straightforward coupling of the gaming clients and servers with a Web and database back end along with HTTP allows for an elegant, simple, yet powerful MMO game architecture. Conceptually, it might help to think of the architecture as composed of two overarching components: the gaming server and client providing MOG functions, and the additional Web and database back end that adds MMO game functions.
--------------------------------------------------------------------------------
Back to top
MMO game architecture implementation
Starship IBM?By explaining the architecture using a familiar game theme of a sci-fi space setting, we can avoid cumbersome terminology such as "initial spawn area game server" and "cloned side mission game server A." Instead, such terms as mothership server and alien outpost server are used in the Starship IBM example game. Imagine a fictitious MMO type game called Starship IBM. It is in a science fiction space setting consisting of:
The mothership "Starship IBM," which is the home ship where players initially enter when they log in to the game. Players can explore the ship and interact with other crew members (players). From there, players can visit other planets to perform missions and return to the mothership after completing a mission.
An alien outpost overrun by hostile aliens. Players can form small "away teams" with other players and travel to this planet to fight the alien monsters.
An unexplored planet, which players can explore in a vehicle.
Using the imaginary game as an example, Figure 3 shows an MMO game architecture based on an actual IBM MMO game.
Figure 3. MMO game architecture implementation
The objects in Figure 3 are:
Client 1...N
The user client machines.
TGEA gaming client
The game client installed on the user's machine.
Mothership TGEA gaming server
The mothership game server is where a player initially logs in when they join the game. From here, players can then play the alien outpost and unexplored planet missions.
There are two mothership game servers on two separate physical servers. There are only a certain number of concurrent clients a server can handle due to the finite amount of server processing power and bandwidth. The additional cloned mothership server was added to handle more clients.
Alien outpost
TGEA gaming server. Serves up the alien outpost missions. Players can form small teams and "travel" to this server on a mission from the mothership.
There are 10 alien outpost game servers on each physical server. TGEA allows for multiple instances of TGEA servers per processor, with each server listening on a unique port for TGEA client connections. Starship IBM is designed so that each alien outpost gaming server runs a mission with a maximum of eight players per mission.
Unexplored planet
TGEA gaming server. From the mothership, players can travel to this server (planet), and explore the world in a vehicle.
Secure WebSphere server
Handles all the HTTP GET requests from the TGEA gaming servers.
Public WebSphere server
Handles all the HTTP GET requests from the TGEA clients. Because the requests are publicly accessible, only secure, appropriate functions should be exposed.
For example, any requests that would make an update to the database should not be exposed. (This function belongs in the secure WebSphere server).
DB2
Stores the game-related data.
Firewall
The hosting environment firewall, which allows access to the gaming servers using the TGEA networking protocol. Allows HTTP requests to access the public WebSphere server.
IBM BladeCenter® system
The physical server running the various gaming, WebSphere, and DB2 servers.
The TGEA server requires Windows.
TGEA client-and-server networking
The game networking connection. Each TGEA client connects to a particular TGEA server. TGEA allows the client to disconnect and then connect to a different TGEA server.
HTTP GET
The TGEA clients make HTTP GET requests to the public WebSphere server, while only the TGEA servers make requests to the secure WebSphere server.
JDBC connection
The connection between the WebSphere servers and DB2.
The architecture in Figure 3 is a sharded architecture, which means that scalability is provided by distributing servers onto multiple physical servers. In this case, the gaming servers are distributed onto six physical servers. So theoretically, they handle six times the number of concurrent users as a single physical server.
Looking at the architecture in Figure 3, you might wonder what number of concurrent players the gaming system can handle. There is no set answer. By default, a TGEA server is configured to handle a maximum of 64 players, but this configuration can be easily updated to any number. At some point, though, when a certain number of concurrent users are on a game server, performance will degrade to the point of making the game unplayable. There is no set limit to this number. Performance is dependent on a number of factors, such as:
The processing power and bandwidth available to a server.
The design of the world. A game consisting of simple avatars walking around a featureless plain, devoid of any objects, in a thick fog, can perform better than a highly interactive first-person shooter set in a richly detailed world full of numerous objects.
The game type. A virtual world or turn-based MMORPG does not require the networking fidelity and performance of a first-person shooter game.
It has been our experience in IBM that approximately 200 concurrent players can be accommodated on a single TGEA server for an MMORPG or virtual world game. Again, this largely depends on the factors mentioned above. Using this approximate number, the MMO architecture in Figure 3 can handle a maximum of:
Two mothership servers = 400 players.
20 alien outpost servers, limited to 8 players each = 160 players.
2 unexplored planet servers = 400 players.
Therefore, the maximum total would equal 960 players.
--------------------------------------------------------------------------------
Back to top
Summary
In this article you learned about the MMO game architecture design based on the IBM MMO game PowerUp. You walked through the architecture of an example sci-fi space game.
Stay tuned for Part 2, which will explore the technical details of the architecture: how the game server and clients and back-end WebSphere and DB2 servers are integrated, and what MMO-type functions are provided by the back end. Explore how traveling from the mothership to the alien outpost is accomplished, how game-related data (such as points) are stored, and what functions are provided by the back end to manage the gaming servers.
Part 3 will assess whether the architecture met the initial goals and will examine future capabilities and limitations.
Overall goals
2010年10月16日 星期六
Designing a 'Single Server' MMORPG
from: http://www.gamesetwatch.com/2009/05/opinion_designing_a_single_ser.php
[In a new opinion piece, designer and Divide by Zero Games founder James Portnow looks at design approaches to unsharding MMOs, in order to create more immersive, integrated game worlds.]
We won’t be able to provide meaningful stories in MMOs until we can allow the players to affect the world they play in. Currently, this is impossible because of the nature of “shards”, “mirrored worlds”, or, as they are best known, “servers” (though this last term is somewhat inaccurate).
The reason for this is: If players on Server A can make choices that differ from those made by players on Server B and those choices actually impact the game world, on the development side you soon end up with a fracturing storyline and so much content to develop that you’ll be fast on your way to learning a lot about bankruptcy court.
There are many ways to solve this problem, my favorite of which is a largely instantiated world tied together with an overarching story structure (I know that’s a dense sentence – come to one of my talks, you’ll find out exactly what I mean by it – but, as it doesn’t really affect the rest of the article, I’m not going to steal more of your valuable time by parsing it).
But I’ve been asked time and again by major companies and interested designers how to build an unsharded MMORPG with a single, vast world space that all the players can play in (for some reason people seem to like to do things the hard way). Well today you’ll all get to hear my answer.
Just to be absolutely clear about what we’re discussing, imagine a game like World of Warcraft, but instead of all the players on different servers they all play on one server. I’m sure your designer brain is reeling at all the problems this presents. You’ve probably already spat out this list of issues, but here’s what I see as the major hurdles to overcome if you are to build such a world.
1. Population Density:
a. Getting enough physical space for all of your players to play in.
b. Making sure towns/areas of interest aren’t over populated.
c. Allowing players to traverse quickly enough between areas of interest.
2. Varied Content:
a. Providing all the players something compelling to do.
b. Having a world that feels cohesive to the players.
3. Production
a. Creating the game economically.
b. Ensuring scalability and cost-effective ongoing content creation.
So now that we’ve identified some of the problems, what will you need to make your Unsharded MMORPG? Well first you’re going to need about 100 million dollars. You’re going to be technology heavy, because you’re going to need a robust backend plus some sophisticated world building systems... What can I say, big world MMORPGs aren’t cheap (and building infinitely extensible ones are even more ‘not cheap’, we’ll be doing a lot of tricks to get it down to 100 mil).
Assuming you’ve got that 100 million. What will you need to do with it? Well, first off, you’re going to need an infinitely extensible world. Clearly you can’t build such a thing by hand without going bankrupt, so what can we leverage to help us create this world? Procedural Content and User Generated Content.
World Enough and Time
In order to create the incredibly immense world you’ll be building you need to create a procedural terrain generation system. This system has to be able to create a varied environment on a large but limited set of assets. How varied? Well remember that we’re talking a world that is potentially unending.
At its very least this world should be about one hundred times bigger than the current World of Warcraft world (this calculation was done with some very rough back of a napkin estimates on how much “space” an MMO player needs to thrive in a virtual world -- which were in turn based on the fact that the peak concurrent on a heavily populated WoW server is about 4000 and that a game of this nature would have to be design for peaks of at least 80k, accounting for differing nature of this type of game).
Your system also has to allow you to scale your world and push the edges of the world out as your populous continues to migrate and explore. Additionally it has to generate a cohesive and traversable world.
There are lots of articles out there on how to procedurally generate terrain, so I’ll skip the specifics. Let’s see what else we need to do.
Works of Days and Hands
Okay, so now we have a world. Now we need to put some stuff in it. Let’s talk about what ends up in an MMO world, because really there isn’t that much, there’s:
1. Cities and Settlements
2. Monsters
3. Natural Resources
4. Dungeons (instances)
Town Building:
Well, first let’s discuss cities and settlements. For the most part we aren’t going to want to build these either…so let’s let the players do it. This leads to a more dynamic world with more player ownership. It also means that the towns will naturally sift out to be where the players need them in your procedurally generated world. Of course this means building a robust and, more importantly, fun, town building/management system. It also means giving players a reason to build and manage towns. Looking at EVE Online’s station management system here is a good guide.
EVE has taught us that player built locations can encourage social interaction, drive player conflict and take some of the burden off developers. How did they do it? By properly incentivizing station building. Rather than being an afterthought – as it is in most MMOs that even include it – ‘town’ building in EVE is an integral part of the design. Player who build stations get economic resources otherwise unavailable, thus highly encouraging structure creation (and defense). This is essential if we are to rely on player created towns in our game.
Additionally this sort of town building gives us one further benefit: it broadens our audience and allows for a new form of play. There will be players who do nothing but manage towns and help build the world. Some of these players will end up being some of your best evangelists, rather than simply washing out of an ‘all grind’ game.
So what sort of system should town creation be? Sadly it is going to require building a totally different interface than your standard MMO controls. This is expensive, yes, but not as expensive as building all the towns of the world. The key to this system is that it be simple to use.
I recommend going with a top down “gods eye view” perspective for your town building, this allows players to see all the buildings in an area in relation to one another and allows you to do basic usability enhancements like outlining buildable plots of land. Building buildings should be as simple as having the resources, clicking on what you’d like to build and where you’d like to build it.
For the most part players should be able to build a single building in a town, let it auto-manage itself and receive income based on the usage of that building (with occasional trips back to upgrade it) if so desired.
This should be completely separate from town management (which we’ll talk about below). On the art side, buildings should be based off simple templates with modular segments for player customization (obviously simple re-colorizing is the cheapest and goes a long way but, if you’ve got extra cash in grip, building finishes will really make your world more vibrant).
Now about town management. There should be a system where certain players can manage an entire town or section of larger cities in order to keep things cohesive. For doing so those players should receive income based off the use of all the buildings in their town (can even be a tax system with a neat slider!) which can only be used to upgrade the town and some sort of personal incentive based off the success of their town.
If you really want to mix things up and incentivize player politics, which we’ll talk about below, allow a “graft” system where players controlling towns or areas can skim some of those funds for their personal use.
How players get to control towns is up to you. Depending on your game you might want it to be a conquest system, or democratic elections, or both.
This system will give your MMO a much needed ‘elders game’ and will allow you to have meaningful player interactions at the high end. This is important since in an unsharded world standard high end raid content becomes much less vibrant as it will be much more diffuse, so anything we can do to create interesting high end gameplay without having to handcraft raids will help a great deal.
But before we can examine that we need to give player’s a reason to congregate in certain areas.
Socially Designed Procedural Resource System
Just because much of our game is procedural doesn’t mean that it isn’t a directed experience. We want players to have the best experience possible, so this means creating “smart” procedural systems that take into account more than just mathematics; we need to build systems that account for human behavior. We’ve just talked about the town creation system. Now we have to consider why people build towns.
For the most part humans build towns around trade routes or natural resources with some minor consideration for defensible terrain. In pre-industrial societies, towns located around natural resources tended to grow until slightly past the point where the per capita access to resources was less in the town and surrounding area than it was in the wilderness.
Trade route cities obviously grow and die with the trade routes they exist along. Major economic centers tend to occur at the confluence of multiple trade routes, so long as that location has some access to independent resources and some natural defenses.
So now the question becomes: how do we allow for this behavior with our resource system? Well first off I’d encourage you to have a designer who has a lot more background in how humans choose to congregate than the above paragraph (fascinating topic, I could go on about it for hours, luckily for you, I won’t…), but just using the above brief information, there are several keys:
1. Remember that monsters ARE resources.
2. Have a weighted distribution system that makes "resource lanes" and "resource nodes"
3. Allow for resource thinning over time
First and foremost, I’ve met a few MMO designers who seem to forget that monsters are their number one resource. Players will tend to congregate in good spawn areas. It’s amazing how quickly even marginally advantageous spawn areas get discovered and exploited. This means your spawn placement and natural resource placement systems should be viewed as one holistic system rather than two disparate entities.
Once you’ve got that overarching view of what your resource system should be, how should you implement it in order to best take advantage of natural human grouping instincts?
Well, first off, you’re going to need to view your world as a grid. You’re going to start off with a group of “resource nodes” - heavy resource concentrations - that will form your starting towns. These towns will have to be built by you, the developer, to give the players a place to start from; luckily the town building system you’ve already implemented should make this cheap.
Once you have these nodes, you’ll have to set the content around them. You will basically be creating "newb zones". This will serve as a seed for your procedural system. Your system will take into account the difficulty of the content in the neighboring grid squares in order to place similar content next to it, getting more difficult as it moves away from the initial nodes.
It will also have to take into account the amount of content in neighboring grid squares in order to algorithmically generate channels of resources that guide players to further resource nodes (where you can expect players to create cities).
It’s important to remember that resource spawning takes place over the lifecycle of your game rather than simply being established when you launch the server, so your spawning mechanic should be able to adjust dynamically as your gameworld/player distribution changes.
It needs to be able to factor in social elements such as what the average level of players in the area and how long an area has been player dominated (so that long held territories become more peaceful, encouraging players to explore and disperse).
I won’t take the time here to go into the specific algorithms to do this, but consider your system to be a cellular automata in the vein of Conway’s Game of Life (if you don’t know what I’m talking about you should immediately stop reading this and Wikipedia Conway’s). This should help contextualize your thinking on how to distribute natural resources, monsters and instances.
Monsters and Instances:
Ok, so far we’ve created the world, we have resources distributed around the terrain so as to direct the players to congregate in specific areas and we have a town building system to facilitate their doing so... now we have to contemplate how to make the actual game fun.
Combat in Fantasy MMOs is well established and many cleverer individuals than I have opined on how to improve it, but it’s not unique to an unsharded world, so let’s skip past that part and talk directly monsters and instances. In an unsharded environment we have to look at monsters and instances fundamentally differently than we do now.
Right now the paradigm is to spawn individual ‘mobs’. To keep things feeling fresh in an environment of this size you should instead be spawning encounters. Some simple scripting will allow your battles to feel more vibrant and get more millage out of a limited amount of content so:
Instead of spawing some orcs, spawn “some orcs fighting some elves” or “some orcs attacking a wagon train” or “some orcs attacking a wagon train with some elves defending it”. In these sentences the terms “orcs”, “elves” and “wagon train” are all variables: they can be plugged in with any monster, humanoid or ‘stuff’. This allows you to create more interesting encounters for your players to stumble upon and while repurposing content to keep costs down.
To re-illustrate: running into orcs sitting around a campfire or elves sitting around a campfire is more interesting than running into a randomly spawned pile of orcs or elves, moreover in both of these scenarios we can reuse the campfire without creating any new assets.
This is important because it will give your world the feel of a vibrant, living environment without handcrafting specific encounters for specific zones (which you won’t be able to afford to do in a world of this magnitude). If you layer on top of this a function in your resource system which tries to deploy a similar set of creatures in any specific geographic region (with these creature sets blending into one another on the boarders) you will have created a cohesive world.
Instances will also have to be largely randomly generated. Handcrafted instances will provide too much incentive for players to group in any specific area, so instead you can distribute entrance points to random instances across your world as it is being created.
Each of these “random” instances should be pseudo-random. They should have attributes such as: difficulty (expected level range), environment/theme, monster types encountered, loot types dropped, etc. Most of these attributes should be set by your procedural spawning system.
What Man Has Made of Man
It is growing late and I grow weary (as probably do you, this is like six pages... of reading... words…), so I’ll try to keep the rest of this brief. There are two major boons that unsharding provides
1. Player interactions can have real meaning.
2. Players can affect the world they play in.
These things may at first seem similar, but I assure you, their distinction is important. Player interactions range from political interactions to the economy of the world where as when I say, “Players can affect the world they play in” I mean that you can give player’s real agency over what occurs in your MMO. There is a vital interplay between the two, but they are separate ideas.
It’s important to focus on both of these things because otherwise, why unshard?
This means that:
A. You should focus on the strength of your MMO: the number of players.
B. Create opportunities for your players to affect the world.
Normally I talk about these two points in great detail whenever I discuss unsharding MMOs, but as that would meander into another several pages of text which would not be directly relate to the practical problem of building a single world, shared story space, fantasy MMORPG I’ll leave you to think on these two points and offer simply this:
In a game of this nature you are not going to have the high end raid content a person might expect from a standard MMORPG, so you have to find other ways to encourage top tier player retention. The two ways to do this are to allow for a thriving player society, where player empires rise and fall, trade monopolies are gained and fortunes are made then lost and to allow player deeds to have a permanent effect on story and game play.
The first way mentioned is sort of the high end PVP aspect of your game, and, if I’m going to give you any advice here on it, it’s to write it into your story. Let player guilds and player wars become part of the lore of your land. Let player factions ally with your NPC groups and influence the future of your game. Patch for things they do and let them engrave their names on part of your world. You’ll get higher retention from this than from a thousand well built instances.
The second is your sort of your PVE. It is events, crafted by you the developer, which take in the whole scope of your world and allow players to decide how they want their world to grow. This can be anything from upping the rates on Goblins spawning and having players fight off the “goblin horde” until finally they unlock the power behind it, to turning in medical supplies to stave off a horrible plague.
Of course, to do this well you have to let people ally with the goblins or the dark forces secretly spreading the plague…and those forces can’t be de facto evil. And you have to be willing to patch differently depending on whether the goblins win or forces opposing them. If you do that you give players a chance to say “I was there”. When they pass by the town ruined in the goblin invasion or the plain haunted by those dead of the plague, they’ll have an unprecedented sense of ownership of the world they are in.
If you want a longer diatribe on how to do this, feel free to email me (my email address is at the end), but simply put, if you are going to do this, remember why you went to all this trouble in the first place.
Flights of Angels
Take this, run with it. I hope it does some of you some good. More importantly I hope it gets us a little closer to delivering the kinds of worlds and the kinds of stories that our medium has long promised. If you use it, I would simply ask that you find the time to share something you learn.
[James Portnow is a game designer, formerly of Activision, and now at Divide by Zero Games, where he is also the founder and CCO. He received his master's degree in Entertainment Technology from Carnegie Mellon University.]
[In a new opinion piece, designer and Divide by Zero Games founder James Portnow looks at design approaches to unsharding MMOs, in order to create more immersive, integrated game worlds.]
We won’t be able to provide meaningful stories in MMOs until we can allow the players to affect the world they play in. Currently, this is impossible because of the nature of “shards”, “mirrored worlds”, or, as they are best known, “servers” (though this last term is somewhat inaccurate).
The reason for this is: If players on Server A can make choices that differ from those made by players on Server B and those choices actually impact the game world, on the development side you soon end up with a fracturing storyline and so much content to develop that you’ll be fast on your way to learning a lot about bankruptcy court.
There are many ways to solve this problem, my favorite of which is a largely instantiated world tied together with an overarching story structure (I know that’s a dense sentence – come to one of my talks, you’ll find out exactly what I mean by it – but, as it doesn’t really affect the rest of the article, I’m not going to steal more of your valuable time by parsing it).
But I’ve been asked time and again by major companies and interested designers how to build an unsharded MMORPG with a single, vast world space that all the players can play in (for some reason people seem to like to do things the hard way). Well today you’ll all get to hear my answer.
Just to be absolutely clear about what we’re discussing, imagine a game like World of Warcraft, but instead of all the players on different servers they all play on one server. I’m sure your designer brain is reeling at all the problems this presents. You’ve probably already spat out this list of issues, but here’s what I see as the major hurdles to overcome if you are to build such a world.
1. Population Density:
a. Getting enough physical space for all of your players to play in.
b. Making sure towns/areas of interest aren’t over populated.
c. Allowing players to traverse quickly enough between areas of interest.
2. Varied Content:
a. Providing all the players something compelling to do.
b. Having a world that feels cohesive to the players.
3. Production
a. Creating the game economically.
b. Ensuring scalability and cost-effective ongoing content creation.
So now that we’ve identified some of the problems, what will you need to make your Unsharded MMORPG? Well first you’re going to need about 100 million dollars. You’re going to be technology heavy, because you’re going to need a robust backend plus some sophisticated world building systems... What can I say, big world MMORPGs aren’t cheap (and building infinitely extensible ones are even more ‘not cheap’, we’ll be doing a lot of tricks to get it down to 100 mil).
Assuming you’ve got that 100 million. What will you need to do with it? Well, first off, you’re going to need an infinitely extensible world. Clearly you can’t build such a thing by hand without going bankrupt, so what can we leverage to help us create this world? Procedural Content and User Generated Content.
World Enough and Time
In order to create the incredibly immense world you’ll be building you need to create a procedural terrain generation system. This system has to be able to create a varied environment on a large but limited set of assets. How varied? Well remember that we’re talking a world that is potentially unending.
At its very least this world should be about one hundred times bigger than the current World of Warcraft world (this calculation was done with some very rough back of a napkin estimates on how much “space” an MMO player needs to thrive in a virtual world -- which were in turn based on the fact that the peak concurrent on a heavily populated WoW server is about 4000 and that a game of this nature would have to be design for peaks of at least 80k, accounting for differing nature of this type of game).
Your system also has to allow you to scale your world and push the edges of the world out as your populous continues to migrate and explore. Additionally it has to generate a cohesive and traversable world.
There are lots of articles out there on how to procedurally generate terrain, so I’ll skip the specifics. Let’s see what else we need to do.
Works of Days and Hands
Okay, so now we have a world. Now we need to put some stuff in it. Let’s talk about what ends up in an MMO world, because really there isn’t that much, there’s:
1. Cities and Settlements
2. Monsters
3. Natural Resources
4. Dungeons (instances)
Town Building:
Well, first let’s discuss cities and settlements. For the most part we aren’t going to want to build these either…so let’s let the players do it. This leads to a more dynamic world with more player ownership. It also means that the towns will naturally sift out to be where the players need them in your procedurally generated world. Of course this means building a robust and, more importantly, fun, town building/management system. It also means giving players a reason to build and manage towns. Looking at EVE Online’s station management system here is a good guide.
EVE has taught us that player built locations can encourage social interaction, drive player conflict and take some of the burden off developers. How did they do it? By properly incentivizing station building. Rather than being an afterthought – as it is in most MMOs that even include it – ‘town’ building in EVE is an integral part of the design. Player who build stations get economic resources otherwise unavailable, thus highly encouraging structure creation (and defense). This is essential if we are to rely on player created towns in our game.
Additionally this sort of town building gives us one further benefit: it broadens our audience and allows for a new form of play. There will be players who do nothing but manage towns and help build the world. Some of these players will end up being some of your best evangelists, rather than simply washing out of an ‘all grind’ game.
So what sort of system should town creation be? Sadly it is going to require building a totally different interface than your standard MMO controls. This is expensive, yes, but not as expensive as building all the towns of the world. The key to this system is that it be simple to use.
I recommend going with a top down “gods eye view” perspective for your town building, this allows players to see all the buildings in an area in relation to one another and allows you to do basic usability enhancements like outlining buildable plots of land. Building buildings should be as simple as having the resources, clicking on what you’d like to build and where you’d like to build it.
For the most part players should be able to build a single building in a town, let it auto-manage itself and receive income based on the usage of that building (with occasional trips back to upgrade it) if so desired.
This should be completely separate from town management (which we’ll talk about below). On the art side, buildings should be based off simple templates with modular segments for player customization (obviously simple re-colorizing is the cheapest and goes a long way but, if you’ve got extra cash in grip, building finishes will really make your world more vibrant).
Now about town management. There should be a system where certain players can manage an entire town or section of larger cities in order to keep things cohesive. For doing so those players should receive income based off the use of all the buildings in their town (can even be a tax system with a neat slider!) which can only be used to upgrade the town and some sort of personal incentive based off the success of their town.
If you really want to mix things up and incentivize player politics, which we’ll talk about below, allow a “graft” system where players controlling towns or areas can skim some of those funds for their personal use.
How players get to control towns is up to you. Depending on your game you might want it to be a conquest system, or democratic elections, or both.
This system will give your MMO a much needed ‘elders game’ and will allow you to have meaningful player interactions at the high end. This is important since in an unsharded world standard high end raid content becomes much less vibrant as it will be much more diffuse, so anything we can do to create interesting high end gameplay without having to handcraft raids will help a great deal.
But before we can examine that we need to give player’s a reason to congregate in certain areas.
Socially Designed Procedural Resource System
Just because much of our game is procedural doesn’t mean that it isn’t a directed experience. We want players to have the best experience possible, so this means creating “smart” procedural systems that take into account more than just mathematics; we need to build systems that account for human behavior. We’ve just talked about the town creation system. Now we have to consider why people build towns.
For the most part humans build towns around trade routes or natural resources with some minor consideration for defensible terrain. In pre-industrial societies, towns located around natural resources tended to grow until slightly past the point where the per capita access to resources was less in the town and surrounding area than it was in the wilderness.
Trade route cities obviously grow and die with the trade routes they exist along. Major economic centers tend to occur at the confluence of multiple trade routes, so long as that location has some access to independent resources and some natural defenses.
So now the question becomes: how do we allow for this behavior with our resource system? Well first off I’d encourage you to have a designer who has a lot more background in how humans choose to congregate than the above paragraph (fascinating topic, I could go on about it for hours, luckily for you, I won’t…), but just using the above brief information, there are several keys:
1. Remember that monsters ARE resources.
2. Have a weighted distribution system that makes "resource lanes" and "resource nodes"
3. Allow for resource thinning over time
First and foremost, I’ve met a few MMO designers who seem to forget that monsters are their number one resource. Players will tend to congregate in good spawn areas. It’s amazing how quickly even marginally advantageous spawn areas get discovered and exploited. This means your spawn placement and natural resource placement systems should be viewed as one holistic system rather than two disparate entities.
Once you’ve got that overarching view of what your resource system should be, how should you implement it in order to best take advantage of natural human grouping instincts?
Well, first off, you’re going to need to view your world as a grid. You’re going to start off with a group of “resource nodes” - heavy resource concentrations - that will form your starting towns. These towns will have to be built by you, the developer, to give the players a place to start from; luckily the town building system you’ve already implemented should make this cheap.
Once you have these nodes, you’ll have to set the content around them. You will basically be creating "newb zones". This will serve as a seed for your procedural system. Your system will take into account the difficulty of the content in the neighboring grid squares in order to place similar content next to it, getting more difficult as it moves away from the initial nodes.
It will also have to take into account the amount of content in neighboring grid squares in order to algorithmically generate channels of resources that guide players to further resource nodes (where you can expect players to create cities).
It’s important to remember that resource spawning takes place over the lifecycle of your game rather than simply being established when you launch the server, so your spawning mechanic should be able to adjust dynamically as your gameworld/player distribution changes.
It needs to be able to factor in social elements such as what the average level of players in the area and how long an area has been player dominated (so that long held territories become more peaceful, encouraging players to explore and disperse).
I won’t take the time here to go into the specific algorithms to do this, but consider your system to be a cellular automata in the vein of Conway’s Game of Life (if you don’t know what I’m talking about you should immediately stop reading this and Wikipedia Conway’s). This should help contextualize your thinking on how to distribute natural resources, monsters and instances.
Monsters and Instances:
Ok, so far we’ve created the world, we have resources distributed around the terrain so as to direct the players to congregate in specific areas and we have a town building system to facilitate their doing so... now we have to contemplate how to make the actual game fun.
Combat in Fantasy MMOs is well established and many cleverer individuals than I have opined on how to improve it, but it’s not unique to an unsharded world, so let’s skip past that part and talk directly monsters and instances. In an unsharded environment we have to look at monsters and instances fundamentally differently than we do now.
Right now the paradigm is to spawn individual ‘mobs’. To keep things feeling fresh in an environment of this size you should instead be spawning encounters. Some simple scripting will allow your battles to feel more vibrant and get more millage out of a limited amount of content so:
Instead of spawing some orcs, spawn “some orcs fighting some elves” or “some orcs attacking a wagon train” or “some orcs attacking a wagon train with some elves defending it”. In these sentences the terms “orcs”, “elves” and “wagon train” are all variables: they can be plugged in with any monster, humanoid or ‘stuff’. This allows you to create more interesting encounters for your players to stumble upon and while repurposing content to keep costs down.
To re-illustrate: running into orcs sitting around a campfire or elves sitting around a campfire is more interesting than running into a randomly spawned pile of orcs or elves, moreover in both of these scenarios we can reuse the campfire without creating any new assets.
This is important because it will give your world the feel of a vibrant, living environment without handcrafting specific encounters for specific zones (which you won’t be able to afford to do in a world of this magnitude). If you layer on top of this a function in your resource system which tries to deploy a similar set of creatures in any specific geographic region (with these creature sets blending into one another on the boarders) you will have created a cohesive world.
Instances will also have to be largely randomly generated. Handcrafted instances will provide too much incentive for players to group in any specific area, so instead you can distribute entrance points to random instances across your world as it is being created.
Each of these “random” instances should be pseudo-random. They should have attributes such as: difficulty (expected level range), environment/theme, monster types encountered, loot types dropped, etc. Most of these attributes should be set by your procedural spawning system.
What Man Has Made of Man
It is growing late and I grow weary (as probably do you, this is like six pages... of reading... words…), so I’ll try to keep the rest of this brief. There are two major boons that unsharding provides
1. Player interactions can have real meaning.
2. Players can affect the world they play in.
These things may at first seem similar, but I assure you, their distinction is important. Player interactions range from political interactions to the economy of the world where as when I say, “Players can affect the world they play in” I mean that you can give player’s real agency over what occurs in your MMO. There is a vital interplay between the two, but they are separate ideas.
It’s important to focus on both of these things because otherwise, why unshard?
This means that:
A. You should focus on the strength of your MMO: the number of players.
B. Create opportunities for your players to affect the world.
Normally I talk about these two points in great detail whenever I discuss unsharding MMOs, but as that would meander into another several pages of text which would not be directly relate to the practical problem of building a single world, shared story space, fantasy MMORPG I’ll leave you to think on these two points and offer simply this:
In a game of this nature you are not going to have the high end raid content a person might expect from a standard MMORPG, so you have to find other ways to encourage top tier player retention. The two ways to do this are to allow for a thriving player society, where player empires rise and fall, trade monopolies are gained and fortunes are made then lost and to allow player deeds to have a permanent effect on story and game play.
The first way mentioned is sort of the high end PVP aspect of your game, and, if I’m going to give you any advice here on it, it’s to write it into your story. Let player guilds and player wars become part of the lore of your land. Let player factions ally with your NPC groups and influence the future of your game. Patch for things they do and let them engrave their names on part of your world. You’ll get higher retention from this than from a thousand well built instances.
The second is your sort of your PVE. It is events, crafted by you the developer, which take in the whole scope of your world and allow players to decide how they want their world to grow. This can be anything from upping the rates on Goblins spawning and having players fight off the “goblin horde” until finally they unlock the power behind it, to turning in medical supplies to stave off a horrible plague.
Of course, to do this well you have to let people ally with the goblins or the dark forces secretly spreading the plague…and those forces can’t be de facto evil. And you have to be willing to patch differently depending on whether the goblins win or forces opposing them. If you do that you give players a chance to say “I was there”. When they pass by the town ruined in the goblin invasion or the plain haunted by those dead of the plague, they’ll have an unprecedented sense of ownership of the world they are in.
If you want a longer diatribe on how to do this, feel free to email me (my email address is at the end), but simply put, if you are going to do this, remember why you went to all this trouble in the first place.
Flights of Angels
Take this, run with it. I hope it does some of you some good. More importantly I hope it gets us a little closer to delivering the kinds of worlds and the kinds of stories that our medium has long promised. If you use it, I would simply ask that you find the time to share something you learn.
[James Portnow is a game designer, formerly of Activision, and now at Divide by Zero Games, where he is also the founder and CCO. He received his master's degree in Entertainment Technology from Carnegie Mellon University.]
MMOG Considerations - Business issues
from: http://www.gamedev.net/reference/design/features/mmog/page4.asp
By now I've shown you that MMO projects are pretty complex beasts to plan; combine the work required to create a stable server application with the cost of the hardware for a strong operating platform, and it adds up to a simple truth: MMO games are expensive. Even with that in mind, you may still have the resources and/or backing to go ahead with the project; in which case, I salute you, and present the following issues for your consideration.
Business structure. Once you've developed the MMO game, are you going to run it yourself, or are you going to set up a separate company to do that? It's like a franchise; the new 'firm' is purely dedicated to running the game that you have developed. It's not hard to find ways to flow money from the new firm back into the old - royalty payments would be the obvious choice - and control can be kept by simply owning the new firm. The advantage is that the finances of the game and of the developer are kept separate; if the game is making losses, it doesn't eat into the developer's own finances, only the game runner's.
Maintenance costs. It won't surprise you to know that if you're going to run a load of servers, you need to supply them with electricity, keep them in a building somewhere, and probably keep that building secure. You'll probably also need to employ a few people to keep them running and deal with crashes and crisis; of course, that means you'll need to heat the building, too (though the heat from 30 servers may be enough, if you can persuade your staff to huddle together a bit). So purely to keep the game running in the same state as it was at launch, you're already running a small office. If you're lucky, your publisher will handle things like subscription payments for you, but you should consider the scenario in which you have to handle that sort of thing yourself. You'd try and automate the actual payment process as much as possible, hopefully to the extent that a problem-free signup requires no human interaction at all; but you'll still need a staff of people on support, to deal with the more severe payment and connection problems.
Content development. As noted before, games which aren't continually revised and updated tend to lose player interest. So how will you go about developing new content for the game? You could keep the team who wrote it in the first place as the people behind the project, but that does keep them from moving on to other projects. You could subcontract. You could hire a permanent development team to keep adding to the game; if you've opted for the 'franchise' approach I suggested earlier, this may be the most attractive option - hire a game designer and crew of artists and programmers to "grow" the game from its initial launch state.
One option which doesn't seem to be commonly used is that of using the player base for content. It's quite possible that there are talented artists or designers in the community; perhaps it would be worth setting up a mechanism whereby they can contribute content they'd like to see in the game?
Revenue stream. It's something of a foregone conclusion that you will want to generate a constant stream of revenue from the game somehow - at the very least to pay for maintenance costs, assuming no new content is being developed. How exactly will you do it? Purely through subscription fees? Will you charge for the initial client portion of the game (the retailed, boxed copy)? How will the subscriptions work - will it be one-fee-fits-all, or different subscription levels for people playing the game to different levels? Will there be 'unlockable content' for players paying more?
There are a few other options to consider. One of the biggest is advertising - is it plausable to try and sell product placements in the game? Unreal Tournament 2004 did this with a fair amount of success. The fantasy setting of many MMO games often makes product placement difficult, but it's still something to be considered.
Middleware. There's already middleware out there aimed specifically at MMO games - Butterfly.net's Butterfly Grid is one example. Such middleware aims to provide both an operating platform and server application framework, upon which you can build your own game without needing to deal with many of the issues raised in this article. There is a certain loss of control, and of course it isn't free, but it's another option that should definitely be considered.
Conclusion
This article has just been a taster of some of the issues facing a developer who wants to enter a project into the MMO market. I've more-or-less sidestepped the game design issues inherent in such games - and yes, there are many further decisions to be made in that area. It is quite telling to note that of all the MMO games out there, only eight or so have more than 50,000 subscribers. (http://pw1.netcom.com/~sirbruce/Subscriptions.html)
Still. While it may be tough work, the MMO genre does provide some very unique opportunities for game designs, most of which have not yet been explored. As the proliferation of broadband technologies continues across the world, the size of the target market increases, and MMO gaming looks like it's going to stick around for quite a while.
By now I've shown you that MMO projects are pretty complex beasts to plan; combine the work required to create a stable server application with the cost of the hardware for a strong operating platform, and it adds up to a simple truth: MMO games are expensive. Even with that in mind, you may still have the resources and/or backing to go ahead with the project; in which case, I salute you, and present the following issues for your consideration.
Business structure. Once you've developed the MMO game, are you going to run it yourself, or are you going to set up a separate company to do that? It's like a franchise; the new 'firm' is purely dedicated to running the game that you have developed. It's not hard to find ways to flow money from the new firm back into the old - royalty payments would be the obvious choice - and control can be kept by simply owning the new firm. The advantage is that the finances of the game and of the developer are kept separate; if the game is making losses, it doesn't eat into the developer's own finances, only the game runner's.
Maintenance costs. It won't surprise you to know that if you're going to run a load of servers, you need to supply them with electricity, keep them in a building somewhere, and probably keep that building secure. You'll probably also need to employ a few people to keep them running and deal with crashes and crisis; of course, that means you'll need to heat the building, too (though the heat from 30 servers may be enough, if you can persuade your staff to huddle together a bit). So purely to keep the game running in the same state as it was at launch, you're already running a small office. If you're lucky, your publisher will handle things like subscription payments for you, but you should consider the scenario in which you have to handle that sort of thing yourself. You'd try and automate the actual payment process as much as possible, hopefully to the extent that a problem-free signup requires no human interaction at all; but you'll still need a staff of people on support, to deal with the more severe payment and connection problems.
Content development. As noted before, games which aren't continually revised and updated tend to lose player interest. So how will you go about developing new content for the game? You could keep the team who wrote it in the first place as the people behind the project, but that does keep them from moving on to other projects. You could subcontract. You could hire a permanent development team to keep adding to the game; if you've opted for the 'franchise' approach I suggested earlier, this may be the most attractive option - hire a game designer and crew of artists and programmers to "grow" the game from its initial launch state.
One option which doesn't seem to be commonly used is that of using the player base for content. It's quite possible that there are talented artists or designers in the community; perhaps it would be worth setting up a mechanism whereby they can contribute content they'd like to see in the game?
Revenue stream. It's something of a foregone conclusion that you will want to generate a constant stream of revenue from the game somehow - at the very least to pay for maintenance costs, assuming no new content is being developed. How exactly will you do it? Purely through subscription fees? Will you charge for the initial client portion of the game (the retailed, boxed copy)? How will the subscriptions work - will it be one-fee-fits-all, or different subscription levels for people playing the game to different levels? Will there be 'unlockable content' for players paying more?
There are a few other options to consider. One of the biggest is advertising - is it plausable to try and sell product placements in the game? Unreal Tournament 2004 did this with a fair amount of success. The fantasy setting of many MMO games often makes product placement difficult, but it's still something to be considered.
Middleware. There's already middleware out there aimed specifically at MMO games - Butterfly.net's Butterfly Grid is one example. Such middleware aims to provide both an operating platform and server application framework, upon which you can build your own game without needing to deal with many of the issues raised in this article. There is a certain loss of control, and of course it isn't free, but it's another option that should definitely be considered.
Conclusion
This article has just been a taster of some of the issues facing a developer who wants to enter a project into the MMO market. I've more-or-less sidestepped the game design issues inherent in such games - and yes, there are many further decisions to be made in that area. It is quite telling to note that of all the MMO games out there, only eight or so have more than 50,000 subscribers. (http://pw1.netcom.com/~sirbruce/Subscriptions.html)
Still. While it may be tough work, the MMO genre does provide some very unique opportunities for game designs, most of which have not yet been explored. As the proliferation of broadband technologies continues across the world, the size of the target market increases, and MMO gaming looks like it's going to stick around for quite a while.
MMOG Considerations - Technology issues
from: http://www.gamedev.net/reference/design/features/mmog/page3.asp
In this section I'll discuss some of the MMOG-specific technology issues, covering both client and server applications (the "server application" being the collection of programs you run on the server-side operating platform).
Caching. Caching is the practice of storing resources that you're more likely to need in a place that's faster to get to than the regular data storage. In MMO gaming, the client programs will pretty much always be caching data - that's because the "regular data storage" is all the way over the network at the server, so accessing it is likely to be very slow. So, we're talking about data that the client will need for the very next frame - models, textures, sounds, and other assets for the player's current status and location would be an example. A key concept with caching is "temporal coherence" - the idea that something is the same from one frame to the next. If something is temporally coherent, there's no point re-loading the data from the server, so you cache it. Beyond that, though, what do you want to cache? How much of that content needs to be picked up dynamically - from the servers - and how much can be included in the client software (i.e. on the CD)? The rule of thumb is: cache anything you might need in a hurry, especially if you can't predict when you'll need it. Graphics for an in-game menu would be an example - you have no way of knowing when the player will bring up the in-game menu, and you definitely don't have time to sit around downloading the graphic from the server. Given that they're unlikely to change, they'd be prime candidates for including in the client software.
At the server end, you'll probably find advantages to caching things as well. You might want to cache things like level geometry around spawn points, so that it can be sent more quickly to players joining the game, rather than having to pull it out of hard storage; or maybe you'll cache information about a specific item that people keep asking for. The best way to find out what to cache is to monitor the requests and see which ones come up the most frequently; a truly rock-solid caching system will do exactly that, and has the advantage that it can adjust over time (i.e. if people hardly ever use a given spawn point, it can detect that and stop caching it).
You'll also want individual servers to cache information within the operating platform. Say you're processing collision detection for a bunch of entities; you'll get a good performance boost from fetching them all into the cache and testing them there, rather than fetching them one-by-one, testing, and then dropping them so that you have to get them from the database again next time.
Also, at both ends of the wire, How large will caches be? Client-side, you'll probably be limited by both the amount of system memory and the amount of hard-disk space you can reasonably use; server-side, though, you've got more control. You should definitely take cache sizes into account when designing your operating platform.
Dynamic Updates. The ability to dynamically update the game - adding new content, removing existing content, or changing content - is crucial. New content keeps the game from getting boring and thus keeps the players playing; there may also be bugs or gameplay balance issues which haven't emerged until several thousand players are present in the game (the latter is especially common). What's your procedure for updating the server application? How about the clients? Will you just stop all the servers completely, make the changes, and relaunch it? Perhaps you'll take down servers one by one - after moving their load off to other machines - update them, and then bring them back into an operational state? That means that until you've done all the machines, some parts of your network will be running the 'old' version and some the 'new' - how will you deal with that? Or perhaps you'll avoid bringing the machines down at all, providing a mechanism for the server application to 'assimilate' changes while it's still running?
The effects of content updates can be extremely extensive. Say you want to create a brand new building in a specific place - will you just sit there and wait until the space you want becomes clear of players? If you're placing the building in a town, that could be quite a wait. So perhaps you'd go for some system which lets you 'mark' the space as about-to-be-built-upon, turning it into a no-entry zone for players until you're done building? The addition of a brand new chunk of geometry to the world would force players in the vicinity to download it from the server into their caches - how would you give the clients 'advance warning' so that they could pick up the mesh before it's needed, to prevent an awkward pause when you place it?
Then, say you want to remove that building a few months later. What do you do with players (or NPCs, for that matter) who are currently inside it? Just drop them on the floor outside with no warning? Would you tell clients that the assets for the building no longer need to be cached, and if so, how?
The most difficult type of update is usually a code update - where you're changing the actual software running either client-side or server-side. You may well want players to refrain from playing until they've installed the client-side update (for example, if it closes an exploit that some players were using to cheat with) - how will you enforce that? Will it be done from within the game, or will players have to exit and visit your game's website to get the update?
Security. It's probably been said that computer security is all about being openly paranoid about your users; if it hasn't, then I'm saying it, because it's true. Any user is a potential cheater, cracker, or malicious script kiddie aiming to take down your servers or generally ruin the game for other people. As such, you should never trust anything information coming from the user - even if you have authentication measures in place to try and ensure that data from the user really is from your client app. Say that your client app is repeatedly sending the position of the player to the server, and the server is simply copying this position into the database as holy writ. An enterprising cracker could modify the client application to send a position value of his choice; and, presto, he's granted himself the ability to 'teleport.' Even if your authentication system prevents modifications to the client software, there are still ways of modifying packets on the fly. If someone finds a way around your authentication, then it's as good as non-existent. So look at things from that point of view, of the position of having no client authentication.
The most secure situation would be to send data to the server purely as a stream of input commands, and have the server do the actual processing of those commands to move the player around and so forth. Even then a cheater could still take advantage of you; he could record a stream of keypresses and send them all to the server in quick succession, allowing him to "press keys" faster than a real human actually could. So you'd still need some kind of 'sanity checker' to watch the user input streams, just to check that what it's seeing is reasonable input for a human player. Of course, this whole approach isn't used because of the processing power it requires; you'd effectively be moving the load from all of your clients (who may number in the hundreds of thousands) onto the server, which is a pretty significant load. The MMO client-server setup represents a pretty significant distributed computing platform, but you have to be careful about what the clients are actually doing.
As such, it never hurts to keep things fairly tightly locked down on the client side; server-side security systems are all very well, but it's nicer if you don't have to use them. Pirating an MMOG client is usually pointless (you still need to pay a subscription fee) so copy-protection schemes are generally unnecessary; instead, the client should be watching out for modified data files (perhaps through a checksumming mechanism), unsavoury-looking programs in memory (particularly so-called 'trainers' that attempt to alter the client's memory while it's running), and debuggers. All three could be signs of someone trying to cheat the game.
Synchronization. Here it is again, though in more of a gaming context - you need to keep all your clients, and all your servers, synchronized (or as close to it as possible). That doesn't just mean that they need to all display the same game clock time; they all need to be working with the same set of data. A player shooting at an enemy will be less than impressed if that enemy actually moved away several minutes ago and they're shooting a ghost (but couldn't tell because the server isn't keeping them up to date).
This issue is particularly prominent in MMO games where different servers are handling different regions of the world. When the player travels from one region to the other, and hits a 'server boundary' (i.e. reaches the edge of the area controlled by the current server), he must transfer to a different server. If the two servers are out of sync, it could cause major problems; for example, standing in one server the player sees a tank rolling in his direction but still far-off; when he switches servers, he discovers that the tank is actually right on top of him. Even to an observer it can look very strange - an entity can be seen to 'skip' boundaries as an out-of-sync position is suddenly updated to a correct one.
In this section I'll discuss some of the MMOG-specific technology issues, covering both client and server applications (the "server application" being the collection of programs you run on the server-side operating platform).
Caching. Caching is the practice of storing resources that you're more likely to need in a place that's faster to get to than the regular data storage. In MMO gaming, the client programs will pretty much always be caching data - that's because the "regular data storage" is all the way over the network at the server, so accessing it is likely to be very slow. So, we're talking about data that the client will need for the very next frame - models, textures, sounds, and other assets for the player's current status and location would be an example. A key concept with caching is "temporal coherence" - the idea that something is the same from one frame to the next. If something is temporally coherent, there's no point re-loading the data from the server, so you cache it. Beyond that, though, what do you want to cache? How much of that content needs to be picked up dynamically - from the servers - and how much can be included in the client software (i.e. on the CD)? The rule of thumb is: cache anything you might need in a hurry, especially if you can't predict when you'll need it. Graphics for an in-game menu would be an example - you have no way of knowing when the player will bring up the in-game menu, and you definitely don't have time to sit around downloading the graphic from the server. Given that they're unlikely to change, they'd be prime candidates for including in the client software.
At the server end, you'll probably find advantages to caching things as well. You might want to cache things like level geometry around spawn points, so that it can be sent more quickly to players joining the game, rather than having to pull it out of hard storage; or maybe you'll cache information about a specific item that people keep asking for. The best way to find out what to cache is to monitor the requests and see which ones come up the most frequently; a truly rock-solid caching system will do exactly that, and has the advantage that it can adjust over time (i.e. if people hardly ever use a given spawn point, it can detect that and stop caching it).
You'll also want individual servers to cache information within the operating platform. Say you're processing collision detection for a bunch of entities; you'll get a good performance boost from fetching them all into the cache and testing them there, rather than fetching them one-by-one, testing, and then dropping them so that you have to get them from the database again next time.
Also, at both ends of the wire, How large will caches be? Client-side, you'll probably be limited by both the amount of system memory and the amount of hard-disk space you can reasonably use; server-side, though, you've got more control. You should definitely take cache sizes into account when designing your operating platform.
Dynamic Updates. The ability to dynamically update the game - adding new content, removing existing content, or changing content - is crucial. New content keeps the game from getting boring and thus keeps the players playing; there may also be bugs or gameplay balance issues which haven't emerged until several thousand players are present in the game (the latter is especially common). What's your procedure for updating the server application? How about the clients? Will you just stop all the servers completely, make the changes, and relaunch it? Perhaps you'll take down servers one by one - after moving their load off to other machines - update them, and then bring them back into an operational state? That means that until you've done all the machines, some parts of your network will be running the 'old' version and some the 'new' - how will you deal with that? Or perhaps you'll avoid bringing the machines down at all, providing a mechanism for the server application to 'assimilate' changes while it's still running?
The effects of content updates can be extremely extensive. Say you want to create a brand new building in a specific place - will you just sit there and wait until the space you want becomes clear of players? If you're placing the building in a town, that could be quite a wait. So perhaps you'd go for some system which lets you 'mark' the space as about-to-be-built-upon, turning it into a no-entry zone for players until you're done building? The addition of a brand new chunk of geometry to the world would force players in the vicinity to download it from the server into their caches - how would you give the clients 'advance warning' so that they could pick up the mesh before it's needed, to prevent an awkward pause when you place it?
Then, say you want to remove that building a few months later. What do you do with players (or NPCs, for that matter) who are currently inside it? Just drop them on the floor outside with no warning? Would you tell clients that the assets for the building no longer need to be cached, and if so, how?
The most difficult type of update is usually a code update - where you're changing the actual software running either client-side or server-side. You may well want players to refrain from playing until they've installed the client-side update (for example, if it closes an exploit that some players were using to cheat with) - how will you enforce that? Will it be done from within the game, or will players have to exit and visit your game's website to get the update?
Security. It's probably been said that computer security is all about being openly paranoid about your users; if it hasn't, then I'm saying it, because it's true. Any user is a potential cheater, cracker, or malicious script kiddie aiming to take down your servers or generally ruin the game for other people. As such, you should never trust anything information coming from the user - even if you have authentication measures in place to try and ensure that data from the user really is from your client app. Say that your client app is repeatedly sending the position of the player to the server, and the server is simply copying this position into the database as holy writ. An enterprising cracker could modify the client application to send a position value of his choice; and, presto, he's granted himself the ability to 'teleport.' Even if your authentication system prevents modifications to the client software, there are still ways of modifying packets on the fly. If someone finds a way around your authentication, then it's as good as non-existent. So look at things from that point of view, of the position of having no client authentication.
The most secure situation would be to send data to the server purely as a stream of input commands, and have the server do the actual processing of those commands to move the player around and so forth. Even then a cheater could still take advantage of you; he could record a stream of keypresses and send them all to the server in quick succession, allowing him to "press keys" faster than a real human actually could. So you'd still need some kind of 'sanity checker' to watch the user input streams, just to check that what it's seeing is reasonable input for a human player. Of course, this whole approach isn't used because of the processing power it requires; you'd effectively be moving the load from all of your clients (who may number in the hundreds of thousands) onto the server, which is a pretty significant load. The MMO client-server setup represents a pretty significant distributed computing platform, but you have to be careful about what the clients are actually doing.
As such, it never hurts to keep things fairly tightly locked down on the client side; server-side security systems are all very well, but it's nicer if you don't have to use them. Pirating an MMOG client is usually pointless (you still need to pay a subscription fee) so copy-protection schemes are generally unnecessary; instead, the client should be watching out for modified data files (perhaps through a checksumming mechanism), unsavoury-looking programs in memory (particularly so-called 'trainers' that attempt to alter the client's memory while it's running), and debuggers. All three could be signs of someone trying to cheat the game.
Synchronization. Here it is again, though in more of a gaming context - you need to keep all your clients, and all your servers, synchronized (or as close to it as possible). That doesn't just mean that they need to all display the same game clock time; they all need to be working with the same set of data. A player shooting at an enemy will be less than impressed if that enemy actually moved away several minutes ago and they're shooting a ghost (but couldn't tell because the server isn't keeping them up to date).
This issue is particularly prominent in MMO games where different servers are handling different regions of the world. When the player travels from one region to the other, and hits a 'server boundary' (i.e. reaches the edge of the area controlled by the current server), he must transfer to a different server. If the two servers are out of sync, it could cause major problems; for example, standing in one server the player sees a tank rolling in his direction but still far-off; when he switches servers, he discovers that the tank is actually right on top of him. Even to an observer it can look very strange - an entity can be seen to 'skip' boundaries as an out-of-sync position is suddenly updated to a correct one.
MMOG Considerations - Operating Platform Issues
from: http://www.gamedev.net/reference/design/features/mmog/page2.asp
Your 'operating platform' - server-side, at least - is the entire server system. It may well be spread over multiple machines, multiple processes and applications, and even multiple continents. What do you need to think about when designing the operating platform for your server?
Load. How many players do you expect a given element in the operating platform to have to handle at any one time? How much spare capacity do you want to build into the system - expecting and thus designing for 10,000 players in the beginning is fine, but if you later on want to expand your market share and thus attract new players, you're going to need to either have the capacity to handle them in the beginning, or come up with a concrete plan for extending the platform at a later point in time. (By a "concrete plan" I do mean something pretty specific; "buy another server" is not very good, while a detailed breakdown of what you will need, step-by-step instructions for integration into the platform, and how much you expect it to increase capacity by, is better). Also, be aware of the difference between total players and concurrent players - even if you have 25,000 total subscribers to your game, you may only have 3,000 players online at any one time. That may not affect the requirements for some parts of the system, but it can heavily affect others.
Distribution. How are you going to spread the load over multiple servers? Will you have one server per subsystem, or will all servers run all subsystems for some part of the game? (For example: will you have one server dedicated to physics processing and another dedicated to AI, or will both servers run physics and AI for separate parts of the world)? How will load distribution be controlled - will you use a single, central 'depot' server, or will individual servers 'pass off' excess work to each other?
Security. You have to provide a way for the outside world to connect to your servers; otherwise, none of your clients could ever log in. But you must be careful; every open port is another potential avenue of attack on your platform. Take the utmost caution when dealing with data from the client; your aim is to make it impossible for someone to crash your servers by sending in a carefully crafted packet. What measures will you take to verify the integrity of the data you're getting, or to protect routines that work with that data? Also, you need to protect your internal services - things like database servers - by keeping them on a network completely separated from the Internet; how will you architect your network to allow the necessary machines to communicate with them, without providing a route to them from the outside world?
Redundancy. Say you have a hardware failure, a power cut, or your security fails and someone manages to crash your servers. How will you go about restoring the world to a state as close as possible to before the problem? You can go for backups; dumping the entire world database to a tape drive every day may be enough for your purposes (and besides, there may be reasons beyond technology failures for keeping backups - such as responding to endemic cheating), but how precisely will that be achieved? Which parts of the system will be backed up?
Another thing to consider might be something like RAID - it means a higher initial cost (instead of buying one 120GB disk for the server, you're buying two or three plus a RAID controller) but it means you've got a second layer of defence before you have to go to your backups (if one disk fails, you can respond in a controlled way to replace it while the game continues to run - though you don't want to hang around, because the other disks could go too). If you do go with a RAID array, how many disks should you invest in? Which RAID level do you want? What's your plan of action when the RAID controller reports that one of the disks has failed (given that you usually can't hot-swap disks around)? How will you cope if the entire RAID array fails (the controller blows out)? Even if you do include a RAID system, it will only account for what's on disk and not what's in memory; so how will you restore what was in memory? If you're not even going to try, how will you minimize the data being kept in memory at any given time, and ensure that it gets written to the disk as often as possible without slowing down?
Synchronization. When you've got a system composed of multiple processes communicating with each other, you're bound to run into synchronization issues some time or another. It's particularly a problem for the database servers - what if two machines are accessing the same record at the same time? Do you implement a 'locking' system so that the first one there gets it and the others have to wait or drop out with an error? Perhaps you go for atomic operations? Or just allow it to happen and to hell with the consequences? The last approach might be perfectly valid if you don't think the chances of a given record being used by multiple other processes are high.
Your 'operating platform' - server-side, at least - is the entire server system. It may well be spread over multiple machines, multiple processes and applications, and even multiple continents. What do you need to think about when designing the operating platform for your server?
Load. How many players do you expect a given element in the operating platform to have to handle at any one time? How much spare capacity do you want to build into the system - expecting and thus designing for 10,000 players in the beginning is fine, but if you later on want to expand your market share and thus attract new players, you're going to need to either have the capacity to handle them in the beginning, or come up with a concrete plan for extending the platform at a later point in time. (By a "concrete plan" I do mean something pretty specific; "buy another server" is not very good, while a detailed breakdown of what you will need, step-by-step instructions for integration into the platform, and how much you expect it to increase capacity by, is better). Also, be aware of the difference between total players and concurrent players - even if you have 25,000 total subscribers to your game, you may only have 3,000 players online at any one time. That may not affect the requirements for some parts of the system, but it can heavily affect others.
Distribution. How are you going to spread the load over multiple servers? Will you have one server per subsystem, or will all servers run all subsystems for some part of the game? (For example: will you have one server dedicated to physics processing and another dedicated to AI, or will both servers run physics and AI for separate parts of the world)? How will load distribution be controlled - will you use a single, central 'depot' server, or will individual servers 'pass off' excess work to each other?
Security. You have to provide a way for the outside world to connect to your servers; otherwise, none of your clients could ever log in. But you must be careful; every open port is another potential avenue of attack on your platform. Take the utmost caution when dealing with data from the client; your aim is to make it impossible for someone to crash your servers by sending in a carefully crafted packet. What measures will you take to verify the integrity of the data you're getting, or to protect routines that work with that data? Also, you need to protect your internal services - things like database servers - by keeping them on a network completely separated from the Internet; how will you architect your network to allow the necessary machines to communicate with them, without providing a route to them from the outside world?
Redundancy. Say you have a hardware failure, a power cut, or your security fails and someone manages to crash your servers. How will you go about restoring the world to a state as close as possible to before the problem? You can go for backups; dumping the entire world database to a tape drive every day may be enough for your purposes (and besides, there may be reasons beyond technology failures for keeping backups - such as responding to endemic cheating), but how precisely will that be achieved? Which parts of the system will be backed up?
Another thing to consider might be something like RAID - it means a higher initial cost (instead of buying one 120GB disk for the server, you're buying two or three plus a RAID controller) but it means you've got a second layer of defence before you have to go to your backups (if one disk fails, you can respond in a controlled way to replace it while the game continues to run - though you don't want to hang around, because the other disks could go too). If you do go with a RAID array, how many disks should you invest in? Which RAID level do you want? What's your plan of action when the RAID controller reports that one of the disks has failed (given that you usually can't hot-swap disks around)? How will you cope if the entire RAID array fails (the controller blows out)? Even if you do include a RAID system, it will only account for what's on disk and not what's in memory; so how will you restore what was in memory? If you're not even going to try, how will you minimize the data being kept in memory at any given time, and ensure that it gets written to the disk as often as possible without slowing down?
Synchronization. When you've got a system composed of multiple processes communicating with each other, you're bound to run into synchronization issues some time or another. It's particularly a problem for the database servers - what if two machines are accessing the same record at the same time? Do you implement a 'locking' system so that the first one there gets it and the others have to wait or drop out with an error? Perhaps you go for atomic operations? Or just allow it to happen and to hell with the consequences? The last approach might be perfectly valid if you don't think the chances of a given record being used by multiple other processes are high.
MMOG Considerations
from: http://www.gamedev.net/reference/design/features/mmog/
The genre of Massively Muliplayer Online Games (or MMOGs for short) is a relatively new, but increasingly popular one for game developers. Encouraged by games such as Everquest, Star Wars Galaxies, or Ultima Online, many new developers are inspired to create designs for huge online worlds with never-before-seen levels of interactivity and detail; professional developers may be more drawn by the constant stream of revenue such projects can provide after launch, in contrast to the usual once-in-a-blue-moon payment from the publisher. The market for MMO games is particularly strong in Asia.
But while such games are popular, they are also one of the most complex projects you will ever encounter. Leaving aside the game design issues - creating and balancing a virtual society that can accommodate thousands of players - the practical issues of developing and running an MMO game are extremely challenging to navigate. This article seeks to provide an overview of some of the issues you should consider when thinking about an MMO project; to give you some of the questions you should be asking, as well as a few possible answers.
Servers, sniffers, systems and suits
The issues I will cover can be broken down roughly into four categories, though there is some overlap:
Connection issues: All issues regarding the actual transfer of data between the client and server - security, stability, speed, protocols, and so on.
Operating Platform issues: All issues regarding the architecture of your server platform. Things like load balancing, backups/redundancy, clustering, etc. Much of this category is common to standard Internet applications, and is not game-specific, but is still of the utmost importance.
Technology issues: All issues regarding the architecture of the system you run on the server. This may seem similar to the previous category, but focuses more on game-specific issues, such as cheat-proofing and dynamic updates.
Business issues: All issues regarding the business side of running an MMO game. Hobbyist developers may be tempted to skip this section, reasoning that it's not relevant to them; in fact, the costs and business details are what most frequently cause hobbyist MMO projects to fail.
One category which you may have expected to show up - client issues - is something that I do not feel I need to cover, as the issues inherent in writing a client for an MMO game are pretty similar to those for any other type of network game.
Connection Issues
In any networked game, you need a system for sending data between nodes in the network. Given that we're talking about MMO games - and all MMO games use a client/server model of communication (as opposed to peer-to-peer, which would suffer major security and management issues) - we can assume that we're talking specifically about the connection between the client and the server. What do we need to be aware of when designing this part of the system?
Security. In general, you do not want third parties to be able to screw around with the data that you're sending over the wire. Interception of packets could lead to a victim's game account details - such as the username and password they use to log into the game - being hijacked. A more worrying scenario is the interception of credit card details, in the case of a game which lets players pay their subscription fees from within the game.
Encrypting every packet you send is probably impractical, not to mention largely unnecessary - intercepting a packet which gives a player's current position in the game world may help someone cheat, but the effort required to obtain such a small piece of information makes it unlikely that people would bother with it. Still, it is likely that you would want the facility to transmit data securely, even if you do not make use of it all the time. So how will you provide that? Which pieces of information will be encrypted and which won't?
Authentication. How can you ensure that the packets you're receiving are really from your client software? How will you prevent someone from hacking together their own client software, which allows them to do things you don't want them to? How can you be sure the source IP address is "safe?"
Stability. Players will not be happy if their connection to your server is cutting off every few minutes. Whatever technology you use to connect your servers to the Internet should be very stable; protection against things like Denial of Service attacks is also something you may want to invest in. Running your MMO server on, for example, a home broadband connection which disconnects for five minutes every few hours, is not advised.
Bandwidth. The connection will be handling packets from potentially hundreds of thousands of clients; the realtime nature of most games means that each player will be sending and receiving many packets per unit of time, compared to non-realtime services like websites. You need to have a connection which, as a whole, can handle this much data; several gigabytes per hour is not an unreasonable expectation. It's unlikely that you'll be handling everything through a single pipe.
Latency. Sadly, it's not yet possible to send information to the average client through quantum entanglement; so, the time it takes for your packets to go from server to client is significant. When sending packets across the Internet, your data will be potentially handled by a pretty large range of devices, each of which running under different loads and imbued with different levels of power. The time taken by a packet to reach its destination will vary, and it may not be small. How will you attempt to minimize latency, at least within the parts of the connection you control? How will you deal with packets that take a long time to reach their destination, and packets that arrive out-of-order?
Failure response. What will you do, both at the client and server ends, if the connection drops or packets are lost? This has implications in your game design - particularly dealing with players who conveniently 'drop' from the game just as they're about to be killed violently - but it bears thinking about on the technical end as well. Should the client try and reconnect, or should it wait for the player to instruct it to do so? Should the server give the player a 'grace period' of a few seconds to reconnect, perhaps based on their ping time, or should it treat the disconnection as a proper one immediately? When it comes to the per-packet level, when will you use guaranteed delivery (which causes all lost packets to be automatically resent, but requires extra packets to be sent back to the server for every packet that is received by the client), and when will you use non-guaranteed delivery?
Protocol. What conventions will you use in your networking protocol? (By 'protocol' here, I do not mean TCP or UDP - rather, I mean the custom protocol that you build on top). An example would be strings: you could go with C-style null terminated strings, or you could go with Pascal-style strings where the length is prepended to the string. Will you stick to network byte order (big-endian) for your data?
The genre of Massively Muliplayer Online Games (or MMOGs for short) is a relatively new, but increasingly popular one for game developers. Encouraged by games such as Everquest, Star Wars Galaxies, or Ultima Online, many new developers are inspired to create designs for huge online worlds with never-before-seen levels of interactivity and detail; professional developers may be more drawn by the constant stream of revenue such projects can provide after launch, in contrast to the usual once-in-a-blue-moon payment from the publisher. The market for MMO games is particularly strong in Asia.
But while such games are popular, they are also one of the most complex projects you will ever encounter. Leaving aside the game design issues - creating and balancing a virtual society that can accommodate thousands of players - the practical issues of developing and running an MMO game are extremely challenging to navigate. This article seeks to provide an overview of some of the issues you should consider when thinking about an MMO project; to give you some of the questions you should be asking, as well as a few possible answers.
Servers, sniffers, systems and suits
The issues I will cover can be broken down roughly into four categories, though there is some overlap:
Connection issues: All issues regarding the actual transfer of data between the client and server - security, stability, speed, protocols, and so on.
Operating Platform issues: All issues regarding the architecture of your server platform. Things like load balancing, backups/redundancy, clustering, etc. Much of this category is common to standard Internet applications, and is not game-specific, but is still of the utmost importance.
Technology issues: All issues regarding the architecture of the system you run on the server. This may seem similar to the previous category, but focuses more on game-specific issues, such as cheat-proofing and dynamic updates.
Business issues: All issues regarding the business side of running an MMO game. Hobbyist developers may be tempted to skip this section, reasoning that it's not relevant to them; in fact, the costs and business details are what most frequently cause hobbyist MMO projects to fail.
One category which you may have expected to show up - client issues - is something that I do not feel I need to cover, as the issues inherent in writing a client for an MMO game are pretty similar to those for any other type of network game.
Connection Issues
In any networked game, you need a system for sending data between nodes in the network. Given that we're talking about MMO games - and all MMO games use a client/server model of communication (as opposed to peer-to-peer, which would suffer major security and management issues) - we can assume that we're talking specifically about the connection between the client and the server. What do we need to be aware of when designing this part of the system?
Security. In general, you do not want third parties to be able to screw around with the data that you're sending over the wire. Interception of packets could lead to a victim's game account details - such as the username and password they use to log into the game - being hijacked. A more worrying scenario is the interception of credit card details, in the case of a game which lets players pay their subscription fees from within the game.
Encrypting every packet you send is probably impractical, not to mention largely unnecessary - intercepting a packet which gives a player's current position in the game world may help someone cheat, but the effort required to obtain such a small piece of information makes it unlikely that people would bother with it. Still, it is likely that you would want the facility to transmit data securely, even if you do not make use of it all the time. So how will you provide that? Which pieces of information will be encrypted and which won't?
Authentication. How can you ensure that the packets you're receiving are really from your client software? How will you prevent someone from hacking together their own client software, which allows them to do things you don't want them to? How can you be sure the source IP address is "safe?"
Stability. Players will not be happy if their connection to your server is cutting off every few minutes. Whatever technology you use to connect your servers to the Internet should be very stable; protection against things like Denial of Service attacks is also something you may want to invest in. Running your MMO server on, for example, a home broadband connection which disconnects for five minutes every few hours, is not advised.
Bandwidth. The connection will be handling packets from potentially hundreds of thousands of clients; the realtime nature of most games means that each player will be sending and receiving many packets per unit of time, compared to non-realtime services like websites. You need to have a connection which, as a whole, can handle this much data; several gigabytes per hour is not an unreasonable expectation. It's unlikely that you'll be handling everything through a single pipe.
Latency. Sadly, it's not yet possible to send information to the average client through quantum entanglement; so, the time it takes for your packets to go from server to client is significant. When sending packets across the Internet, your data will be potentially handled by a pretty large range of devices, each of which running under different loads and imbued with different levels of power. The time taken by a packet to reach its destination will vary, and it may not be small. How will you attempt to minimize latency, at least within the parts of the connection you control? How will you deal with packets that take a long time to reach their destination, and packets that arrive out-of-order?
Failure response. What will you do, both at the client and server ends, if the connection drops or packets are lost? This has implications in your game design - particularly dealing with players who conveniently 'drop' from the game just as they're about to be killed violently - but it bears thinking about on the technical end as well. Should the client try and reconnect, or should it wait for the player to instruct it to do so? Should the server give the player a 'grace period' of a few seconds to reconnect, perhaps based on their ping time, or should it treat the disconnection as a proper one immediately? When it comes to the per-packet level, when will you use guaranteed delivery (which causes all lost packets to be automatically resent, but requires extra packets to be sent back to the server for every packet that is received by the client), and when will you use non-guaranteed delivery?
Protocol. What conventions will you use in your networking protocol? (By 'protocol' here, I do not mean TCP or UDP - rather, I mean the custom protocol that you build on top). An example would be strings: you could go with C-style null terminated strings, or you could go with Pascal-style strings where the length is prepended to the string. Will you stick to network byte order (big-endian) for your data?
The Inner Workings of an MMO Server Farm
from: http://lotrovault.ign.com/View.php?view=CommunityArticles.Detail&id=11
This discussions here is not about me but suffice to say I have had plenty of experience with MMO server clusters, infrastructure design and loadtesting to be able to provide some experienced insight on the subject to folks debating the release of LoTR release setup. I will caveat my post here with the tidbit that each MMO engine is a little (if not a lot) different than others and so although this informations pertains to MMO engines I have worked with it is in no way specific or directly relational to LoTR. Take it for what it is or leave it. :)
MMOs require many many PC boxes to operate a single "server". These clusters of boxes are often called a server farm. In your server farm you can have as few as 3-4 servers right up to 20-30 (or in the case of WoW, 80+) servers. Truely the number is unlimited but for network management purposes going beyond these numbers in a cluster makes for a logistics and backbone nightmare.
Inside your server farm there are usually 3-4 distinct tasks that are divided up aming the servers. Patching, authentication, database and world/zone servers are a pretty standard jobset. The patching servers are usually nothing more than a customized FTP/HTTP server that allow connections from a game client which is then updated as required using checksums to validate the necessary client files on the patch server. The authentication server is pretty much just that. A server that takes the user name and password from game clients and authenticates the account against the entries in the database, verifies the account is active and flags the clients account for access to their player data. Database servers are... well.. database servers.. duh :) They are customized to optimize the datastream between the database and the auth/world servers for player data parsing. The world/zone servers are the boxes that handle the actual player data for specific areas of the game world, including load blancing, as needed, for multiple boxes to handle a single area.
Where all this becomes important is when a company underestimates or overestimates their world's population numbers and is unprepared to deal with the fallout for that. As an example, in MMOs the authentication and patcher servers are generally the hardest hit. As MMOs have grown in size, it is not uncommon for a game client to see massive (even over 1 gig) patches at release to make adjustments (that probably should have been done in beta *sigh*) to the game. PErsonally I dont anticipate a "massive" patch for LoTR due to that fact that yes, eventhough they need some tweaks and adjustments, the game is in a very polished state and I don't forsee any sweeping changes. Even IF a game should reuire massive patches for hundreds of thousands of clients simultaneously though the great part is these patch servers dont have any effect on game server performance because they have zero interaction with the databases or the world servers. This means that players ingame arent having their datastream jammed up by 600k people trying to patch 500MB each into their game client.
Authentication servers are the next lot of servers to get hammered. Every single player trying to connect to the game has to bounce through an Auth server in order for the game to verify their username and password and pass back the appropriate player selection info for their account. The great part about auth servers is they run a VERY simple process that is just a checksum of UID/PWD data against the account database and pass back a small stream to the client. Sure the auth servers can back up but when most database servers today can process thousands (if not tens of thousands) of reuests per second it's kind of hard to imagine a company having to deal with more than 50 thousand login attempts per second and causing authentication issues.
World servers and Zone servers are the heart of every MMO. This is also the area that most people are very clueless about the workings of an MMO. World servers generally take up the lions share of the actual server farm. These are the boxes that share the duties of maintaining world object data for their specific area or landblock. The beauty of technology today is that load balancing server power with today's blade platforms means that as a servers load becomes unmanageable the MMO engine can "recruit" more servers dynamically, "on demand" so to speak, within seconds.
Think of a cluster of world servers under "anticipated" traffic. Lets use an example of 10 servers slugging away happily with average load.. In the wings there are another 20 servers, configured and running just idling away with no traffic telling the master server they are there and waiting for instructions. Then it happens!!!! AMG a Gandalf sighting in section 13B!!!! All of the sudden hundreds of players swarm into section 13B to hail the mighty white wizard and the poor server(s) running that section begin to panic. They reach their predetermined "performance" threshhold and send a panic'd scream for help to the shard's master server. After some gentle reassurance to the beseiged world server the master server reaches into the overflow server pool and assigns backup servers A, B and C to alieviate some of the load from sector 13B. Thirty seconds into the great lagfest that was sector 13B the overflow servers step in and all of the sudden the lag becomes more bearable.. all of the sudden the 500 hobbits charging the brandywine bridge trying to head off Gandalf the Gray are actually able to move again and the world is a better place..
Configuring servers the first time is tough. I've been there :) Optimizing the servers and software for maximum data throughput and performance levels takes time but once it is done, replication is accomplished in minutes / hours not in days or weeks. Having a server farm of xxx servers, a good network team can configure the appropriate database/world servers in hours. If release is a raging success and they STILL end up swamping their entire overflow pool of servers it's a matter of mere hours to load their software and configure NEW DB and world servers once they are made available from the service provider.
Does all of this mean Turbine will have a flawless launch? Of course not :) Turbine isn't famous for making all the right decisions that I can certainly attest to. But is the fact they may have to deal with 500k to 1.5 million beta players any reason to scream Doom Doom! I highly doubt it. For me that fact that there is so much intrest in the game can only mean good things ahead for LoTR.
If I had to call it I'd say that definitely open beta will be crowded. I am sure of that. I would rathe rit be crowded for the first 4 weeks and settle into medium player levels after that than have that fiasco of a launch that was DDO where after 4 weeks the servers are ghost towns and you cant find anyone to play with because they had too many servers at launch.
Anyways this was just some insight on how MMO server farms are setup. Hope it was a fun read.
This discussions here is not about me but suffice to say I have had plenty of experience with MMO server clusters, infrastructure design and loadtesting to be able to provide some experienced insight on the subject to folks debating the release of LoTR release setup. I will caveat my post here with the tidbit that each MMO engine is a little (if not a lot) different than others and so although this informations pertains to MMO engines I have worked with it is in no way specific or directly relational to LoTR. Take it for what it is or leave it. :)
MMOs require many many PC boxes to operate a single "server". These clusters of boxes are often called a server farm. In your server farm you can have as few as 3-4 servers right up to 20-30 (or in the case of WoW, 80+) servers. Truely the number is unlimited but for network management purposes going beyond these numbers in a cluster makes for a logistics and backbone nightmare.
Inside your server farm there are usually 3-4 distinct tasks that are divided up aming the servers. Patching, authentication, database and world/zone servers are a pretty standard jobset. The patching servers are usually nothing more than a customized FTP/HTTP server that allow connections from a game client which is then updated as required using checksums to validate the necessary client files on the patch server. The authentication server is pretty much just that. A server that takes the user name and password from game clients and authenticates the account against the entries in the database, verifies the account is active and flags the clients account for access to their player data. Database servers are... well.. database servers.. duh :) They are customized to optimize the datastream between the database and the auth/world servers for player data parsing. The world/zone servers are the boxes that handle the actual player data for specific areas of the game world, including load blancing, as needed, for multiple boxes to handle a single area.
Where all this becomes important is when a company underestimates or overestimates their world's population numbers and is unprepared to deal with the fallout for that. As an example, in MMOs the authentication and patcher servers are generally the hardest hit. As MMOs have grown in size, it is not uncommon for a game client to see massive (even over 1 gig) patches at release to make adjustments (that probably should have been done in beta *sigh*) to the game. PErsonally I dont anticipate a "massive" patch for LoTR due to that fact that yes, eventhough they need some tweaks and adjustments, the game is in a very polished state and I don't forsee any sweeping changes. Even IF a game should reuire massive patches for hundreds of thousands of clients simultaneously though the great part is these patch servers dont have any effect on game server performance because they have zero interaction with the databases or the world servers. This means that players ingame arent having their datastream jammed up by 600k people trying to patch 500MB each into their game client.
Authentication servers are the next lot of servers to get hammered. Every single player trying to connect to the game has to bounce through an Auth server in order for the game to verify their username and password and pass back the appropriate player selection info for their account. The great part about auth servers is they run a VERY simple process that is just a checksum of UID/PWD data against the account database and pass back a small stream to the client. Sure the auth servers can back up but when most database servers today can process thousands (if not tens of thousands) of reuests per second it's kind of hard to imagine a company having to deal with more than 50 thousand login attempts per second and causing authentication issues.
World servers and Zone servers are the heart of every MMO. This is also the area that most people are very clueless about the workings of an MMO. World servers generally take up the lions share of the actual server farm. These are the boxes that share the duties of maintaining world object data for their specific area or landblock. The beauty of technology today is that load balancing server power with today's blade platforms means that as a servers load becomes unmanageable the MMO engine can "recruit" more servers dynamically, "on demand" so to speak, within seconds.
Think of a cluster of world servers under "anticipated" traffic. Lets use an example of 10 servers slugging away happily with average load.. In the wings there are another 20 servers, configured and running just idling away with no traffic telling the master server they are there and waiting for instructions. Then it happens!!!! AMG a Gandalf sighting in section 13B!!!! All of the sudden hundreds of players swarm into section 13B to hail the mighty white wizard and the poor server(s) running that section begin to panic. They reach their predetermined "performance" threshhold and send a panic'd scream for help to the shard's master server. After some gentle reassurance to the beseiged world server the master server reaches into the overflow server pool and assigns backup servers A, B and C to alieviate some of the load from sector 13B. Thirty seconds into the great lagfest that was sector 13B the overflow servers step in and all of the sudden the lag becomes more bearable.. all of the sudden the 500 hobbits charging the brandywine bridge trying to head off Gandalf the Gray are actually able to move again and the world is a better place..
Configuring servers the first time is tough. I've been there :) Optimizing the servers and software for maximum data throughput and performance levels takes time but once it is done, replication is accomplished in minutes / hours not in days or weeks. Having a server farm of xxx servers, a good network team can configure the appropriate database/world servers in hours. If release is a raging success and they STILL end up swamping their entire overflow pool of servers it's a matter of mere hours to load their software and configure NEW DB and world servers once they are made available from the service provider.
Does all of this mean Turbine will have a flawless launch? Of course not :) Turbine isn't famous for making all the right decisions that I can certainly attest to. But is the fact they may have to deal with 500k to 1.5 million beta players any reason to scream Doom Doom! I highly doubt it. For me that fact that there is so much intrest in the game can only mean good things ahead for LoTR.
If I had to call it I'd say that definitely open beta will be crowded. I am sure of that. I would rathe rit be crowded for the first 4 weeks and settle into medium player levels after that than have that fiasco of a launch that was DDO where after 4 weeks the servers are ghost towns and you cant find anyone to play with because they had too many servers at launch.
Anyways this was just some insight on how MMO server farms are setup. Hope it was a fun read.
訂閱：
文章 (Atom)