Rock on, I'm an open source contributor!
A while back (October 2008), I was working on a high-performance search indexing app at a previous job. We were originally going to write it in C/C++ (which was the smart thing for that particular project), but were pressured to do it MUCH faster than the original (incomplete and poorly-written) scope called for, so we jumped to Python.
It was my first actual Python project, so I ended up giving myself a VERY fast education on the basics of the language and jumping immediately into more advanced stuff — including modifying C extensions to Python to handle different types than they were designed for.
Specifically, the Berkeley DB bindings did not support multiple keys being associated with individual entries, which did not bode well for indexing, say, articles with multiple keywords. It took one string as an optional index. Berkeley is a very low-level data store; we chose it because it was (and probably still is) the most blazingly, ridiculously fast storage engine out there that also handles concurrent connections well.
Anyway… I patched it to take either a single string (as it already did) or a list of strings (which the C bindings supported but the Python layer didn’t yet).
For a long time I didn’t think the changes had made it in, but upon closer inspection, they did:
Search the page for “Kung Phu,” which is how I’m credited. I’m pretty satisfied!
Rendering woes
Yeesh.
Since I’m running the client itself in a single process, anything that has to go from one to the other has to be serializable, since it’s being run through pipes… well, that means I can’t really use some of the pygame types in my basic game objects.
So I’ve started a rendering ‘engine’ that’s pretty much just a set of wrapper classes; each one takes a certain game object (maps and panels right now, I think they’re generic enough that it won’t grow much beyond that), updates its internals with sprites as necessary, and provides update/draw methods (they’re extending pygame.sprite.Group for this purpose).
The problem is that these rendering objects aren’t persistent… they’re regenerated every frame. This might cause performance issues, but the thing I’m more concerned with is animation… I might have to track the position of an actor within the actor model itself rather than within a rendering object, which I didn’t particularly want to do, but if that’s what I can depend on being persistent, I might just have to settle for that little breach of my arbitrary protocol.
I’m not too sure I like the way this whole client method is going anyway… I guess we’ll see. Damn you, Unladen Swallow, hurry up and kill the GIL so I can just use threads!
Diversions and leveling discussion (was: Argh!)
Well, I haven’t just failed to update this blog, I’ve failed to work on the project for quite a while.
I’ve been working with Django at length… and while doing that I’m improving my Python knowledge bit by bit. At the moment I’m a little hesitant to really dive back into the game until I’ve finished Pro Django, which is an absolutely awesome read (though I’m only a quarter of the way through it). Picking up on some of the really interesting topics it covers — particularly metaclasses and Python introspection — which I have not had the opportunity to use before, having been a Mild-Mannered (PHP) Web Programmer for such a long time.
Anyway, I’ve also been reading up on various people’s thoughts on replacing the traditional RPG level system (i.e. kill things -> get experience -> gain a level -> get a cookie). While I’m quite used to that and enjoy its simple, linear progression, I do think it’d be interesting to either use something different or augment it to be a little more flexible and enable/encourage non-combat means of advancement (I’m a HUGE fan of crafting).
While taking in a discussion of How to Replace Levels in MMO’s, I started to put together some of my own initial thoughts (which I sketched out early on in design, a terribly long while ago) and see if I could perhaps take some inspiration from Mr. Green’s points.
I am still a big fan of the experience -> level paradigm -> stat paradigm; to me it’s a nice, clear way to see a little number increase (i.e. the cookie) from your work. He discusses a system used in LotR Online, which reminds me a bit of the Evility system in Disgaea 3 — basically, you complete a rather mundane quest/task and you receive a trait for it. You can then enable a set number of traits to affect your character’s abilities/stats… at least, that’s how I understood it.
This seems like a great idea; I think I’m going to plan on using this (or something similar) as a secondary leveling system. Well, I guess it’ll kind of be tertiary…
What I’ve been considering for a long time and am pretty set on is having several (perhaps many) ‘level’ stats for all characters… your combat level, your leathercrafting level, your potion brewing level, etc. Rather than just having these sit idly by waiting for you to use them, however, I want them all to contribute, most likely in one of these manners:
- Generate your overall level based on all skill levels, and generate your base stats from your overall level
- Have each skill contribute to a certain set of your base stats based on its level; either do away with total level or generate it approximately (perhaps as an average)
The main goal here is to partially or completely eliminate the need for combat if a player doesn’t want that particular grind. If you love crafting and can manage to do it until your crafting level is through the roof, then I think your total stats should reflect it enough to enable you to take on other tasks — whether this means combat, another craft, or anything that will depend on your ‘level’ — without basically starting over.
Say a level 100 crafter (this is just an example; I don’t want to cap at 100) wants to go fight something. By this point he’s probably made some bank in game and can afford more than a wooden sword… should he have to take his gold-plated mail and wield his +2 axe of smiting against… blue slimes? Or should his crafting levels make him effectively, say, a level 10 fighter, so he can go after gold golems? The latter is what I’m after, and I really think it’ll add a lot for people who want more than a little role play with their hack ‘n’ slash.
Tools and environment
Since I have not had the chance to touch the code (my wonderful girlfriend is in town over the weekend; it was a very good trade), I figure that as long as she was surfing her knitting blogs, I would regurgitate some old info, interspersed with new stuff, into one entry so it’s easily accessible.
Environment
Both client and server are being written in Python (you may have guessed this due to the prominent associated tag spread throughout the previous entries). The client uses pygame for rendering.
Client and server are also using the multiprocessing library, which allows me to run what would normally be threads (limited by Python’s global interpreter lock so that only one could actually do anything at a time), sharing (serializable) resources between processes, chief among them being the map. On the server, this will be instrumental in handling client connections outside of the main game thread.
While I had started with C++ (and am still quite the fanboy), Python is immensely faster for development. Many things that require questionable libraries in C++ are either built directly into the core of Python or supported by well-maintained extensions to the language.
Tools
The server will only ever run on Linux; to start with, it will live on my EC2 instance. That makes server development relatively simple; since I work and play in Linux, it’s already right at home.
The client is also being built in Linux; however, the intention has always been to build it for Windows as well (hopefully Mac, too, but definitely Windows). For this reason, I’m trying to stick with completely portable libraries; I think I’ve been successful so far.
As far as actual code, I write mine in a great little text editor called Geany. While I’ve been pushed towards Komodo by some very capable coworkers, it’s not in Portage (Gentoo’s package manager), so I stubbornly abstain. ;) Running the code is handled directly from the command line.
Motivations
This project is meant to satisfy a few cravings on my part:
- A good, extensible online 2D RPG
- A solid, elegant Python game engine
- Proof that creating an MMO doesn’t have to take a team
- Proof that creating an MMO doesn’t have to cost a mint
- Learning the heck out of Python
Now that I’ve conspicuously added the ‘MM’ to my ‘O,’ yes, I do think this will scale well enough to handle arbitrary numbers of players.
I’m not using a separate data store for game world objects, so I don’t have a connection bottleneck there; since I’m running separate processes for each connection to the server, the GIL isn’t a problem either. My main considerations, then, are how many processes a single server can handle effectively, and how much game information it can store in memory.
Since the multiprocessing library can almost trivially expand across multiple machines, I can at least attempt to scale up in this way; also, I have experience integrating with memcached, which could assuage my single-machine memory limitation. Both of these things introduce latency (inter-server communication, in both cases); how much will depend upon how well I can limit the necessity for connection processes to access and manipulate game data, and how efficiently I can represent the game world and its contents in memory.
Part of the reason it took me so long to pick up Python is that I was always under the impression that it was slow… as in slower than PHP. Yes, it is slower than reasonably well-written C and C++; however, it is a great deal faster than most interpreted languages. The execution speed argument is meaningless on the client; any machine built in the last five years will almost certainly run the client application with no trouble at all.
The server is obviously a different matter. I could almost certainly handle a notably greater number of connections with a good C++ server app than with one written in Python… however, while my C++ is good, it’s not exceptional — and, quite honestly, if I tried it in C++ again I would probably just never finish it. That’s not a general argument for Python, of course, but in my case it’s another relevant point.
Anyway, I’m working late tonight on a database migration for a client… going to see about writing a bit on the game before I start on that. Thanks for reading!
Woohoo! Map movement!
Yeah, I know, it doesn’t look like much… ;)
This is just a unit test to ensure that movement between map tiles is being handled correctly. Tiles you can walk on are labeled as []; tiles you can’t are marked as //. The tile the actor (player object) is on is marked with the number. This is a really simple way to make sure the map is only allowing the player to walk on the right tiles (earlier on it successfully keeps the player from walking off the map, which is good, but will be handled slightly differently for multiple maps).
It (purposefully) ignores animation and view (the subset of this map that should be displayed on screen); separating these things is something I needed to accomplish anyway, since my previous ‘tech demos’ had them tied together, which is not acceptable for the actual client.
Multiprocess command input
Well, this multiprocessing crap ROCKS, but man does it make things… interesting.
After wrestling with a serialization issue, I realized I can’t keep a lock as an instance variable on an object I need to pass between processes; I should have figure that out sooner, but I had written it to work with threading, which doesn’t have this particular quirk.
Now I’m working on getting a sensible method of passing events out of the input thread and determining where best to process them; I knew it’d be a little hairy, but it looks like I either made some questionable decisions early on about where to implement things or just did so with the (now forgotten) intention of moving them later on, because I’m left wondering whether actor movement should really be handled by actors themselves or by the maps they are associated with, and how much graphical information the actor should keep about itself — it needs to track its own sprite sheet, but as far as rendering, I think I need to introduce a generic processor in the GUI thread to handle this.
So at this point I’m largely working on juggling code to put things in places that make sense, with the objective of successfully passing commands from the input thread to the GUI thread in a way that correctly and efficiently invokes the required functionality, and (when necessary) passes it back to the server as well.
Multiprocessing is a go!
After a whole lot of on-and-off frustration with this problem, I FINALLY figured this one out (after reading the same documentation about thirty times, but oh well).
So now I’ll be able to handle processes not only on the server, which is pretty much necessary for keeping many concurrent users happy, but also on the client, so that I can run graphics, networking, and input handling in separate processes.
I had written them in threads before; while that was fine and all, Python (the C implementation, anyway, which is the standard) has a global interpreter lock, meaning that even if you’re running multiple threads, only one can ever actually be executing at a time. Which really sucks if you want actual concurrency, since, you know, that doesn’t qualify at all.
Next step: Folding this into the current client code! It’ll require a little reworking architecturally, but even with that, soon I’ll be able to start putting my GUI and input code back in the mix together.
Exciting stuff!
Switching to Python and other intriguing developments
After getting a bit tired of my non-interactive HR/blockquote updates page, I installed a portal and reopened the forum as the main site.
Python/Pygame
I’ve switched from C++ to Python for all code and from straight SDL to Pygame for graphics. The increase in productivity per hour is awesome.
Client progress
Separately, I have implemented both sprite motion and movement control from inputs and animation from the same. However, I haven’t mixed them yet, and I’m still going back and forth on threading vs. multiprocess for the client, which is where I’m a little stalled right now (largely trying to find a Python process library that handles shared objects gracefully).
Server progress
The server accepts and responds to connections and messages. Right now the client and server both gzip their packets; I’ll probably encrypt them first as well, just because I am a security nut job. The server isn’t terribly complex yet and hasn’t been the focus of development.
Technical Tidbits
Animation, while a slight pain in the ass, turned out to be pretty reasonable (and relatively inexpensive in terms of resources as I’m using sprite sheets and only ever keeping that image in memory, which will make it relatively simple to get the paper doll effect going when I get to that point; yes, that will definitely be part of the game).
Milestone
0.1
Genotype
* User can interface with client through game pad.
* Client encodes and sends requests to server.
* Server receives, decodes and interprets requests.
Phenotype
+ Client can move around and between maps.
+ Client can activate objects on map.
+ Client can view and navigate menus.
The only genotype (back end) item that is not complete is game pad integration, which should be relatively simple with Pygame’s joystick classes; front-end behavior (phenotype) is much more complicated. While moving around a single screen has been implemented, it’s not yet worked into the main client code.
This ‘milestone’ turned out to be… probably a good 30% of the actual work involved in writing the game, so breaking it down further once I had a better idea of how it would be handled would have been a good idea. I’m using trac to build task lists for myself, though, which is accomplishing the same end in a more useful manner.
Stay tuned! I’m hoping to have the multiprocessing library solidly in hand in the next week, and once that happens I will be integrating player motion back into the client and building the map infrastructure.