How do I diagram ping between servers on the Net?

[Jeff]

You do realize that every every server is broadcasting all its traffic to every other server its the equivalent of every server getting blasted with all the traffic?

Whether or nto this busy you anything really then comes down to whether the processing that creates/responds to that traffic is very heavy or not comapred to the cost of handling the packets.

If it is then such a scheme may help you increase the number of players you can handle. If the handling the traffic is teh bottleneck though then you will not be much better off then one sevre handlign everything.

(In fact, sinxe it takes more processing to do net packetr handling then to communicate between processes i na common memory space you may even be worse off.)

Also, have you considered what happens to your total simulation if some of the nodes fail?

Welcome to the complexities of multi-processing for massive scalability.

[Jeff]

Btw...

Multicast reduces the bandwidt hrequriemenst on the wire, but the wire usually isnt your bottelneck.  It doiesnt reduce your load on your machines handling the traffic theyr evieve, in fact it increases it over point to point communication.

[blahblahblahh]

PS what net protocol are you intending to use? IP doesn't support multicast until v6 (which is still being used by almost nobody outside of universities and research depts ).

Games published by our own members! Check 'em out!

[jc]

If some of the nodes fail, then the total simulation will continue. However, clients on the failed servers will appear to the distributed network's servers' clients to have quit and logged off the distributed network. An attack on a server will not affect a distributed network.

I originally intended to implement the Java Reliable Multicast Service. However, I now suspect that the only two solutions to a seamless, scalable world exist: more hardware and bandwidth, or less servers and clients.

I am out of ideas. I thought that the principles of a distributed IRC network could apply to a MMOG, but I was wrong. Apparently, clever coding is not a substitute for more hardware and bandwidth.

What if I develop a distributed IRC network and, for the client, a 3D graphical front-end? If a front-end allows a client to move around, gesture, and walk from room to room, is the system robust enough to support a MMOG? After all, Efnet has over 100 servers and more than 20,000 users!

[Mark Thornton]

PS what net protocol are you intending to use? IP doesn't support multicast until v6 (which is still being used by almost nobody outside of universities and research depts ).

Multicast seems to work with class D (IP4) addresses. Getting a publicly usable address and getting ISPs to propagate the broadcasts may be another matter.

[blahblahblahh]

Getting a publicly usable address and getting ISPs to propagate the broadcasts may be another matter.

Tis what I meant ... in practice, IPv4 has no multicast. In practice, IPv6 should have it (although I've not checked the current status of in-the-wild IPv6 nets for quite some time, so YMMV)

[blahblahblahh]

I originally intended to implement the Java Reliable Multicast Service. However, I now suspect that the only two solutions to a seamless, scalable world exist: more hardware and bandwidth, or less servers and clients.

I am out of ideas. I thought that the principles of a distributed IRC network could apply to a MMOG, but I was wrong. Apparently, clever coding is not a substitute for more hardware and bandwidth.

That's a silly conclusion (and incorrect). All we've said is that naive algorithms are not going to solve the problem, and that you have to put in a great deal more work and "clever coding" Sorry, that's just the way it is. Realistically, if you can come along with a trivial architecture and solve fundamentally hard problems that for 30+ years no-one else has solved without months and months of design work then maybe you're a genius .

What if I develop a distributed IRC network and, for the client, a 3D graphical front-end? If a front-end allows a client to move around, gesture, and walk from room to room, is the system robust enough to support a MMOG? After all, Efnet has over 100 servers and more than 20,000 users!

Did you read the article I quoted? Every prospective customer that comes to us we ask to see a Game-Concept before we talk to them - it's rarely possible to give meaningful answers to any specific questions unless there is a game design already on the table.

Your question above is a classic example. If your game design is "IRC with avatars, no more than 5 avatars to a room, 10 second delay to move from room to room" then yes of course the answer is yes.

If your game design is for a standard MMORPG, then of course the answer is "snowball's chance in hell".

[cknoll]

... in practice, IPv4 has no multicast. In practice, IPv6 should have it (although I've not checked the current status of in-the-wild IPv6 nets for quite some time, so YMMV)

I wrote an applicaiton that sent information using multicast, and this was in 1994, so IPv4 indeed does have multicast even in practice.  I think that if you have control over the servers and the network that joins them (and most of the big hitters in the MMO arena do have this) multicast is definitely an option, considering you will be able to configure the routers that interconnect the servers to forward multicast packets.  In my experience with multicast, it is EXTREMELY scalable in the sense that if you need to notify 100 servers of 1k of state change, you send 1k of data, and not 100k (1 per sever).  And it remains 1k even as you move up to 1000 servers.  

It has always been my interest to implement a 'distributed simulator' that scaled well to multiple clients, but I'll keep these ideas  to myself because I don't think I could handle blahbhalbhalbhaa's cynicism on it. heh.

Jeff:

Multicast reduces the bandwidt hrequriemenst on the wire, but the wire usually isnt your bottelneck.  It doiesnt reduce your load on your machines handling the traffic theyr evieve, in fact it increases it over point to point communication.

I don't agree with this assessment, but I didn't work at TEN.

-Chris

[blahblahblahh]

It has always been my interest to implement a 'distributed simulator' that scaled well to multiple clients, but I'll keep these ideas  to myself because I don't think I could handle blahbhalbhalbhaa's cynicism on it. heh.

FYI I'm not cynical here, it's just that I see quite a lot of people (especially in the mainstream games industry) embarking on MMOG projects without the vaguest idea of how incredibly huge a topic distributed-systems dev is. Many people assume that dist-sys server programming is not much different from network programming, which is like assuming that doing a modern OpenGL 3D engine with all the eye-candy is not much different from rendering 2D in standard VGA (which you know every grahpics card handles equally well, for a start!).

Jeff: [... Multicast reduces the bandwidt hrequriemenst on the wire, but the wire usually isnt your bottelneck...]

I don't agree with this assessment, but I didn't work at TEN.

It depends very much on application (which seems to be something I'm repeating myself on ).

In any complex distributed system, you have about 5 (can be many more) major simultaneous bottlenecks and every time you improve one by a factor of e.g. 2, your performance improves by a factor of 1.1 because you've just become limited by a different one of the five. You can play whack-a-mole like this for a loooooong time without showing much improvement.

So, bw could be your current bottleneck, and then you multicast and it disappears, but there's no improvement because it was masking not just one but four other major ones. One part of the problem is that people often forget how much the system is holistic, i.e. normally you'd think that handling packets was a tiny amount of CPU power, but in a dist system it could be enough to push you over the edge into a chain reaction where throughput is slightly less than incoming requests - after 1 second, everything's a little slow, after 10 it's getting very slow, after 1 minute the server is fatally overloaded.

[jc]

Did you read the article I quoted? Every prospective customer that comes to us we ask to see a Game-Concept before we talk to them - it's rarely possible to give meaningful answers to any specific questions unless there is a game design already on the table.

NAME OF GAME
CubeWorld is the name of our MMOG.

DESIGN HISTORY
CubeWorld is the successor to strikeNET, our attempt of yesteryear to design a MMOG. strikeNET underwent a plethora of revisions, from a MMOFPS in the vein of PlanetSide, to a MMORPG that combined elements of Grand Theft Auto: Vice City, Splinter Cell: Pandora Tomorrow, and Hitman: Contracts into a simulation of the world of espionage.

strikeNET was an attempt to appeal to the mainstream, but we realized that no game could appeal to everyone. All games restrict freedom through goals and boundaries. strikeNET was to be a MMOG to end all MMOGs. strikeNET  promised a world in which users could do anything, without bounds.

Although we wanted that world to be one of espionage in the current Cold War, we knew that strikeNET lacked the ambition to deliver a world with no rules. We wanted a sandbox in which thousands could collaborate to create their own worlds in a single, massive universe. CubeWorld was a result of our odium for conventional MMOGs.

GAME OVERVIEW
A multi-user domain, or MUD, is a server that hosts a web of rooms in which clients interact with clients, items, and rooms. Clients can create items, rooms, and links to rooms. Interaction between rooms is impossible because a MUD is zoned, not seamless. Furthermore, items in a MUD serve no function.

CubeWorld is a sandbox in which thousands can collaborate to create their own worlds in a single, massive universe. CubeWorld allows you to build whatever, whenever, wherever. You can script anything, from a battlefield in which soldiers fight for power, for glory, forever, to a city in which anyone can be an assassin.

FEATURE SET
CubeWorld stresses simplicity, to the point that CubeWorld resembles Quake III: Arena. In Quake III: Arena, a master server maintains a list of servers and ping. Each client queries the master server to join a server. Each server maintains its clients, items, and buildings. If a server disappears, then its clients, items, and buildings disappear.

In CubeWorld, a master server maintains a list of servers and ping. A client can query the master server for a list of servers and ping. However, she maintains an account on her favorite server only. Her account maintains her avatar, items, and buildings, which persist after she logs off.

CubeWorld comprises a distributed network. Donors can donate servers to the distributed network. Each server adds avatars, items, and buildings to the universe that exists on the distributed network. If a server disappears, then its avatars, items, and buildings disappear from the universe that exists on the distributed network.

Each server sends multicast packets of avatars, items, and buildings to every other server. Each packet includes a header that declares where in the universe the packet occurred. If a server analyzes a packet header and discovers that the packet does not affect its avatars, items, or buildings, then the server ignores the rest of the packet.

In CubeWorld, your avatars, items, and buildings are easy to edit. If a model is animated, then it consists of geometric primitives around the bones of a skeletal animation. If a model is unanimated, then it consists of geometric primitives. Each primitive is a solid color instead of a texture.

CubeWorld is friendly to modems. A user can download a client in no time, because the distributed network streams all models and sounds to the client. A model in the form of geometric primitives around the bones of a skeletal animation requires little bandwidth. A sound in the form of Ogg Vorbis also requires little bandwidth.

THE GAME WORLD
In CubeWorld, the universe is a massive, seamless grid. Clients can cordon off plots of land on the grid to create buildings with geometric primitives. A group of clients can share access to a plot of land, which allows collaboration in real-time. Clients can also use a scripting language, similar to UnrealScript, to bring their plots of land to life.

Games published by our own members! Check 'em out!

[blahblahblahh]

That's very helpful in letting us know what you're trying to do, thanks.

The first thing I'd suggest is that you look at Neverwinter Nights, which seems to be almost identical except for the fact that it has fancy graphics instead of primitives, and that the transfer from server to server is less integrated.

In terms of a game, I cannot stress how much better off you would be actually inventing a single game rather than trying to create CubeWorld as a world/toolkit. Your observations about strikenet were good - it's a very bad idea to try and "just make a world where people can do anything" (it tends to be incredibly boring, and you'll never finish the project - this has happened to very many people already!). But then, instead of taking the wisdom from that, you seem to be about to make some of the same mistakes again?

Or, to put it another way, you are not trying to write a game - CubeWorld is a toolkit, not a game. There is no "fun" and there is nothing to do. This is fine - so long as you don't delude yourself into thinking that you are writing a game when you are not. I would suggest you consider making an MMOG with some fun gameplay first...

[jc]

CubeWorld is similar to Second Life in that Second Life allows avatars to move around a seamless landmass, create items, use a scripting language to bring their creations to life, and cordon off plots of land to create buildings. Second Life blurs the line between games and toolkits, but Second Life is a MMOG.

[blahblahblahh]

CubeWorld is similar to Second Life in that Second Life allows avatars to move around a seamless landmass, create items, use a scripting language to bring their creations to life, and cordon off plots of land to create buildings. Second Life blurs the line between games and toolkits, but Second Life is a MMOG.

There.com had $57 million funding and still a lot of people find it "too boring" to play.

Really, these "games" are just chatrooms with extra features. They may look very pretty, but they don't have that special something that defines a game - fun.

One criticism of designs such as yours for Cubeworld is that it is hard to distinguish them from:

• we give people computers
  
• and a programming language
  
• and a programming manual
  
• and expect them to be grateful
  

This is all, of course, just personal opinion. I think you might want to attach slightly more importance to it than that because I've been in the industry for 6 years now in one form or another, and because I've seen countless people propose your "game" over and over again and none of them have ever got around to actually making anything that was fun. Shrug; maybe you will be different, or maybe you don't mind that you're not making a game...but I can assure you that what you're proposing is a lot *harder* than making a game, once you factor in the difficulty of getting people to "play" it.

[cknoll]

JC-
 I'm missing the part of your project description that addresses the 'G' part of MMOG (that would be, the actual Game).  When thinking about the game, what are the rules? What are the goals?  What are the obsticles?  What are the rewards?  What are the penalties?  Most importantly (since it's MM), how can these goals be accomplished, obsticles overcome, through use of teamwork? etc.

I believe thinking in terms of single player is a good start to determine what you want to 'do' within the game space, but eventually you need to add the mutli-player dimension to the mix and see how it comes together.  The project you describe has interesting technical aspects to it, but from a game perspective, it's a little hollow.

-Chris

[Jeff]

Some comments.. FWIW:

(1) NWN has great client and map building technology.  Its server system sucks, is incredibly naive,  and can handle up to about 16 players before totally fallign over if yo uare very lucky and on a very fast machine.

(2) There are indeed distributed solutions to some of these problems. They are not easy or obvious and JUST throwing the problem across  more computers and saying "now its DESTRIBUTED so its fault tolerant and scalable" doesn't work.  

"Clever coding" alone won't get you to the right answers.  There are entire text books on destributed simulation.  If you are serious about this area then I'd suggest you start there.  If you are going to be coming to GDC then stop by the Sun booth and I can show you some other interesting things.