Re: [pygame] Re: pygame and file object compatibility; slow.

[Brian Fisher <brian@xxxxxxxxxxxxxxxxxxx>]

What about the font loading segfaulting? Are you still getting that issue? If so, what version of pygame are you using on what OS?


In case you are curious, the problem with reading from tar files as file-like objects is that with tar files, seeking to a certain point in the stream is expensive cause tar files don't have any index of the tar contents (you have to read each files header and seek past it in the archive to figure out what else is in there) so it has to re-read the tar file from the start to move to a new place in the file. The pygame/SDL code that does the loading doesn't know that, so it tries to seek around, expecting seeking to be cheap. Note that compressed tar (gz, bzip2, etc.) makes the situation much much worse - because once the tar stream is compressed, you can't seek past files, now you have to read and decompress to figure out when you've passed one archived file and found the next ones header. It's pretty much hopeless to try and read your assets out of a compressed tar if you are randomly accessing the asset files.

Zip files are much much better because they have a file index (zip files compress files first, then archive those compressed files with an index), the pygame loader may still be a bit slower on loading from compressed files in zips, and if that becomes an issue, you could try reading the file contents uncompressed out of the zip into a string, and then loading from a StringIO:
http://docs.python.org/library/stringio.html

On Sun, Mar 28, 2010 at 1:50 AM, Maize <vhxonline@xxxxxxxxx> wrote:

After fussing with it for a few hours it seems to be a problem with
either pygame or the tar file objects. The time it takes to extract
the file into memory is around 0.003s or so. The problem occurs
exactly when it hits the pygame.image.load. I eventually found a
method to load them faster (8s as opposed to 9 hours for 379 images.)
Still a bit long.  Anywho, I did move onto ZIP and whala, problems are
gone. Now it only takes 0.07s to load all 379 images. Thanks for the
response though. I think I'll just stick with ZIP for now.