Submitting a bug is... KIND of like contributing... right?
Since I enjoy writing at myself so much, I figure I’ll go ahead and update this! :)
I finally started getting back into this little by little, and actually made some pretty good progress. I fixed the rendering quirks I was running into by digging into and correcting my usage of the multiprocessing library; I still have some locking to implement, but it’s quite trivial compared to the major revamp I just finished.
There’s just a small problem left with rendering on moving to a new map, but I know where the problem is, and it’s something that just hasn’t been fully thought out yet.
As for the title, as I was going through my multiprocessing implementation, I hit a VERY frustrating issue. Long story short, it turns out you can’t start this particular type of shared memory controller in a file that’s imported by the invoked script — whether it’s created in an import or not, is start() method MUST be called from the script you actually RUN. Yeah, screwy. I filed a bug, though, so rock on!
Things that are now working:
Client
- Input, graphics and networking are handled in separate threads
Server
- Processes are spawned for and take input from any number of clients
General
- Movement around a map and between maps works
- Collision detection with map boundaries and impassible tiles works
I’m noticing that the graphics and networking processes are eating an irresponsible amount of CPU time, so that’s going to be something to look into, but I don’t think it’s going to be a big problem; there shouldn’t be anything too non-standard about the way I’m handling those things.
Multiproc woes
Well, I think I’ve figured out the infuriating problems I’ve been butting my head against with multiprocessing.
Turns out I can send and update objects and get them back with the changes intact, and I can send and update lists with the same kind of success… but I can’t send and update a list of objects, update the objects, and expect to get them back with their state updated.
So it looks like my ideal scenario of having a simple execution stack containing (among other things) a map object that tracks its tiles as a list is not going to work. I do think I can still use the stack for menus; if I’m recalling correctly (it was late last night), I can modify an object and update it by copying, modifying and replacing it in its list (which is stupid, inefficient and maddening, but works).
I’d like to see if processing or parallelpython can handle this scenario… but I’ve already burned so much time figuring it out in multiprocessing that I’m not too thrilled with the idea of trying to abandon it.
Either way… I did manage to get the input process passing commands to the top element of the execution stack, it just didn’t work as expected because of multiprocessing’s apparently-undocumented restriction on how many nested levels of data (about… one) you can depend on sharing between processes.
Tools and environment
Since I have not had the chance to touch the code (my wonderful girlfriend is in town over the weekend; it was a very good trade), I figure that as long as she was surfing her knitting blogs, I would regurgitate some old info, interspersed with new stuff, into one entry so it’s easily accessible.
Environment
Both client and server are being written in Python (you may have guessed this due to the prominent associated tag spread throughout the previous entries). The client uses pygame for rendering.
Client and server are also using the multiprocessing library, which allows me to run what would normally be threads (limited by Python’s global interpreter lock so that only one could actually do anything at a time), sharing (serializable) resources between processes, chief among them being the map. On the server, this will be instrumental in handling client connections outside of the main game thread.
While I had started with C++ (and am still quite the fanboy), Python is immensely faster for development. Many things that require questionable libraries in C++ are either built directly into the core of Python or supported by well-maintained extensions to the language.
Tools
The server will only ever run on Linux; to start with, it will live on my EC2 instance. That makes server development relatively simple; since I work and play in Linux, it’s already right at home.
The client is also being built in Linux; however, the intention has always been to build it for Windows as well (hopefully Mac, too, but definitely Windows). For this reason, I’m trying to stick with completely portable libraries; I think I’ve been successful so far.
As far as actual code, I write mine in a great little text editor called Geany. While I’ve been pushed towards Komodo by some very capable coworkers, it’s not in Portage (Gentoo’s package manager), so I stubbornly abstain. ;) Running the code is handled directly from the command line.
Motivations
This project is meant to satisfy a few cravings on my part:
- A good, extensible online 2D RPG
- A solid, elegant Python game engine
- Proof that creating an MMO doesn’t have to take a team
- Proof that creating an MMO doesn’t have to cost a mint
- Learning the heck out of Python
Now that I’ve conspicuously added the ‘MM’ to my ‘O,’ yes, I do think this will scale well enough to handle arbitrary numbers of players.
I’m not using a separate data store for game world objects, so I don’t have a connection bottleneck there; since I’m running separate processes for each connection to the server, the GIL isn’t a problem either. My main considerations, then, are how many processes a single server can handle effectively, and how much game information it can store in memory.
Since the multiprocessing library can almost trivially expand across multiple machines, I can at least attempt to scale up in this way; also, I have experience integrating with memcached, which could assuage my single-machine memory limitation. Both of these things introduce latency (inter-server communication, in both cases); how much will depend upon how well I can limit the necessity for connection processes to access and manipulate game data, and how efficiently I can represent the game world and its contents in memory.
Part of the reason it took me so long to pick up Python is that I was always under the impression that it was slow… as in slower than PHP. Yes, it is slower than reasonably well-written C and C++; however, it is a great deal faster than most interpreted languages. The execution speed argument is meaningless on the client; any machine built in the last five years will almost certainly run the client application with no trouble at all.
The server is obviously a different matter. I could almost certainly handle a notably greater number of connections with a good C++ server app than with one written in Python… however, while my C++ is good, it’s not exceptional — and, quite honestly, if I tried it in C++ again I would probably just never finish it. That’s not a general argument for Python, of course, but in my case it’s another relevant point.
Anyway, I’m working late tonight on a database migration for a client… going to see about writing a bit on the game before I start on that. Thanks for reading!
Multiprocess command input
Well, this multiprocessing crap ROCKS, but man does it make things… interesting.
After wrestling with a serialization issue, I realized I can’t keep a lock as an instance variable on an object I need to pass between processes; I should have figure that out sooner, but I had written it to work with threading, which doesn’t have this particular quirk.
Now I’m working on getting a sensible method of passing events out of the input thread and determining where best to process them; I knew it’d be a little hairy, but it looks like I either made some questionable decisions early on about where to implement things or just did so with the (now forgotten) intention of moving them later on, because I’m left wondering whether actor movement should really be handled by actors themselves or by the maps they are associated with, and how much graphical information the actor should keep about itself — it needs to track its own sprite sheet, but as far as rendering, I think I need to introduce a generic processor in the GUI thread to handle this.
So at this point I’m largely working on juggling code to put things in places that make sense, with the objective of successfully passing commands from the input thread to the GUI thread in a way that correctly and efficiently invokes the required functionality, and (when necessary) passes it back to the server as well.
Multiprocessing is a go!
After a whole lot of on-and-off frustration with this problem, I FINALLY figured this one out (after reading the same documentation about thirty times, but oh well).
So now I’ll be able to handle processes not only on the server, which is pretty much necessary for keeping many concurrent users happy, but also on the client, so that I can run graphics, networking, and input handling in separate processes.
I had written them in threads before; while that was fine and all, Python (the C implementation, anyway, which is the standard) has a global interpreter lock, meaning that even if you’re running multiple threads, only one can ever actually be executing at a time. Which really sucks if you want actual concurrency, since, you know, that doesn’t qualify at all.
Next step: Folding this into the current client code! It’ll require a little reworking architecturally, but even with that, soon I’ll be able to start putting my GUI and input code back in the mix together.
Exciting stuff!