I know a few people who are going to try and cheat at my game (*sniffle*), but I've made the server secure, encrypted packets etc. But I'm lost as to what encryption scheme. I'm currently encrypting using DesEDE with a dynamic key (changing each outgoing packet), and I think this'll sort of murder the server, encrypting so many things so quickly (or trying). I would write my own encryption/descryption methods, but I'm wondering what the most efficient/safest things to do are.
No matter what encryption scheme you use for normal game data, anyone who wants to hack your packets will do so. Take a gander at the emulators and cheat tools out there for UO, EQ, DAOC and whatever else exists. I've followed the development of a DAOC emulator for sometime, and though Mythic changed their encryption scheme a couple of times the emulator team was able to crack it in short order. Account/credit card info is a different story. For that end of the game, you can go with some more heavy duty encryption that mere mortals aren't likely to crack in a lifetime.
Packet Encryption, like CD copy protection, is a barrier against the lazy. My advice is forget about it. Better to spend your time thinking about packet compression schemes (sending as much data in as few bytes as possible). The two primary areas of cheating you need to be concerned with are client side tools and packet hacks - neither of which can be prevented by encryption.
Not long after DAOC was released there were a couple of 'radar' programs available that allowed players to see the position of enemy players who were otherwise out of view. This gives a giant, unfair advantage to radar users. Mythic eventually learned how to detect radar programs and, through a combination of red flag raising at the code level and observation by employees of those so marked, have suspended and/or banned several radar-using accounts.
Programatically, third party tools like the radar program are hard to detect in the general case. One way to handle it is to code the server to look for suspicious activity. What constitutes suspicious activity depends on your game. The problem is that this can become very complex. For example, detecting radar users might involve storing a sampling of a player's position updates over an interval of time, the tracing back through the list of position updates to determine at what point an attacker started on a straight vector toward another player and if the victim was actually visible at the time. If the victim was not visible, increment a counter on the player's account. Once the counter has passed a certain threshold, raise a red flag so that a staff member can then silently observe the player in game. This is just me rambling, but this is the sort of thing you have to deal with to detect third party program use programmatically. And where do you draw the line? Do you try to detect macroers (like those that plagued UO once upon a time)? Radar users? Key bots (something that presses a certain key or a sequence of keys over and over and over without player interaction)? There's no easy answer to this.
Another method of third party tool detection at the code level is to scan the operating system's active processes for known cheat programs. While this can be much simpler and, potentially more accurate, than the general case of detection on the server side, it can also be error prone. It's possible to mistake a legitimate program for a cheat tool, the creators of the cheat tool will most likely be diligent at changing the application's memory signature, window title, or whatever else you used to clue in on it once they discover what you are doing. At best, you would still only be able to raise a red flag and have someone monitor the account. The bad part is that the client would have to send a packet to the server to do so. It wouldn't take much for someone to write a packet sniffer that detects the red flag packet and either discards it or modifies it such that no red flag is raised.
Another form of cheating you need to be on the look out for is packet modification. Unlike detection of third party tools, this is normally much easier to detect in the general case, but it requries a great deal of forethought to do so. First, you need to validate that each packet is the size it should be. You might also want to send a CRC key with each packet to verify integrity (though this, again, is only good against the lazy). Most important of all is that the server needs to strictly validate all data coming from the client. Is it really possible for player X to be running at double the normal speed? Is player Y's hit point value valid? The server knows the game rules, and it knows the game mechanics. With that in mind you should be able to validate every piece of data that comes across the pipe and catch 99.9% of the modified packet values that come through. Usually, when people cheat they do it in a big way. Forget increasing hitpoints by 10, they do it by 1000. 10% speed increase? Bah, try 500%! These will be the majority of cheats you see and will be blatantly obvious. The other 0.1% are the ones that are smart enough to cheat within the bounds of the game mechanics. These are likely impossible to catch in code, and most likely won't give you any reason to red flag them. The best you can hope for with these guys is that they are observed doing something they shouldn't be able to, or make a mistake later on.
Exploiters are another category of cheater, but it's hard to catch them without first knowing what the exploit is. Some of them might get caught in the nets of the third party tool detection, or through packet validation. But the majority will likely be reported by other players.
I hope all of this makes sense to you. Encryption of game data is really a wasted effort for a small scale project, IMO. Big companies like Sony are going to be targetted by every leet script kiddie and his brother, so encryption will discourage a large number of them (though over time the more persistent will figure out the encryption scheme, decode the packets, and make it all available somewhere online in one form or another). If you really, really insist on encrypting your packets, you might want to roll you own algorithm - either something from scratch or a modified version of an existing one. Any encryption libraries that are publicly available to you are also publicly available to the cheaters. By rolling your own you might get lucky and delay them a few days