Assume the string is read from disk, instead of doing a pygame image load. The speed bump truly surprised me!
The problem with switching to that approach (loading your images as encompressed strings off of the disk) is that disk is much much slower than CPU. Todays great transfer rates for disks are like 83MB/s (if you get faster than that, it's out of the disk cache) whil memory buses are like 533MB/s, and the perf difference is much much worse (like 5-15MB/s) if the disk head has to seek (which it often does)
It's hard to really see that effect in testing as well - if you ran on an NT os (like winXP), then if you try and load the same file off of disk it usually comes from a memory cache which grows as big as it needs up to all your free memory. So if you time how long it is to load the same image 100 times or something, you threw out the impact of disk IO.
Additionally, trying to save your images uncompressed makes the disk stuff much worse, cause compression makes them significantly smaller on disk. And the disk is the slowest part.
To validate all this, I wrote a test script that takes an image, and saves enough copies of itself to entirely fill the RAM on my machine (so that it couldn't all be cached, and disk IO would have to happen) both as the original compressed and as a flattened "tostring" version. Then it times loading all that stuff. It also times loading the same image over and over for comparison.
below are the results of the test on my machine: ------------- FINAL RESULTS ------------- cached .png file loaded at 1.0054 seconds per MB or 0.0312 seconds per file cached flat file loaded at 0.0323 seconds per MB or 0.0081 seconds per file flat files are 8.1x the size of the optimized .png .png files loaded at 1.3262 seconds per MB or 0.0412 seconds per file flat files loaded at 0.3432 seconds per MB or 0.0858 seconds per file
So on my box, it's more than twice as fast to load compressed .pngs with image.load than uncompressed files with image.fromstring, even though the cpu and memory work to load compressed files is about 30 times longer than just making the image from a string (image.load does appear to be a real dog...) I think the reason the flat file loading lost was because the increased size of the disk IO (8x as large files) made the flat file loading lose out (the disk load took 10x the time of making the image from the string)
The other thing that is interesting about this test to me, is how much faster the disk reads were with the larger flat files (it loaded 3x as many bytes in the same time). So if you want your loading to go faster, large files are better. So packing your data in some big data.dat is good for loading time from disk (cause then the OS can load and cache large blocks at a time)
I would imagine that the flat file stuff could be faster on many computers, but there is no way I can see it being anywhere near 32x in practice except if you are loading the same images over and over again.
--- script and fle attached
Attachment:
TestFile.png
Description: PNG image
import pygame
import os
import stat
import time
import shutil
import platform
# don't use a cpu time func if we can help it - will be bad!
platform_string = platform.platform(aliased = True, terse = True)
if 'Windows' in platform_string:
timer_func = time.clock
else:
timer_func = time.time
def GetFileSize(filepath):
return os.stat(filepath)[stat.ST_SIZE]/(1024.0*1024.0)
###############################################################33
test_file_name = "TestFile.png"
test_dir = "TestDir/"
if not os.path.isdir(test_dir):
os.mkdir(test_dir)
# some initial set up
image_format = "TestImage%06d.png"
original_image = pygame.image.load(test_file_name)
pngsize = GetFileSize(test_file_name)
desired_size = 512
# create a flattened version
flat_format = "TestFlat%06d.dat"
image_as_string = pygame.image.tostring(original_image, 'RGBA')
image_size = original_image.get_size()
first_flat_file_name = test_dir + flat_format % 0
first_flat_file = file(first_flat_file_name, "wb")
first_flat_file.write(image_as_string)
first_flat_file.close()
flatsize = GetFileSize(first_flat_file_name)
print "sleeping to let disk stuff settle..."
time.sleep(3)
times_reload_same_image = 1000
print "timing loading the same .png file %d times" % times_reload_same_image
start_time = timer_func()
for load_index in xrange(times_reload_same_image):
image = pygame.image.load(test_dir + (image_format % 0))
if (load_index % 100) == 99:
print " loaded %d" % (load_index + 1)
end_time = timer_func()
image_reload_time = end_time - start_time
print "sleeping to let disk stuff settle..."
time.sleep(3)
print "timing loading the same flat file %d times" % times_reload_same_image
start_time = timer_func()
for load_index in xrange(times_reload_same_image):
flat_file = file(test_dir + flat_format % 0, "rb")
flat_data = flat_file.read()
flat_file.close()
image = pygame.image.fromstring(flat_data, image_size, "RGBA")
if (load_index % 100) == 99:
print " loaded %d" % (load_index + 1)
end_time = timer_func()
flat_reload_time = end_time - start_time
# set up enough disk data to exhaust the cache
num_image_copies = int(desired_size/pngsize)
print "Creating %d image copies..." % num_image_copies
for image_index in xrange(num_image_copies):
shutil.copyfile(test_file_name, test_dir + (image_format % image_index))
if (image_index % 100) == 99:
print " created %d" % (image_index + 1)
num_flat_copies = int(desired_size/flatsize)
print "Creating %d flat copies..." % num_flat_copies
for image_index in xrange(1, num_flat_copies):
shutil.copyfile(first_flat_file_name, test_dir + flat_format % image_index)
if (image_index % 100) == 99:
print " created %d" % (image_index + 1)
# now do the serious timings...
print "sleeping to let disk stuff settle..."
time.sleep(10)
print "timing loading %dMB of .png images" % desired_size
start_time = timer_func()
for image_index in xrange(num_image_copies):
image = pygame.image.load(test_dir + (image_format % image_index))
if (image_index % 100) == 99:
print " loaded %d" % (image_index + 1)
end_time = timer_func()
image_load_time = end_time - start_time
print "sleeping to let disk stuff settle..."
time.sleep(10)
print "timing loading %dMB of flat images" % desired_size
start_time = timer_func()
for image_index in xrange(num_flat_copies):
flat_file = file(test_dir + flat_format % image_index, "rb")
flat_data = flat_file.read()
flat_file.close()
image = pygame.image.fromstring(flat_data, image_size, "RGBA")
if (image_index % 100) == 99:
print " loaded %d" % (image_index + 1)
end_time = timer_func()
flat_load_time = end_time - start_time
print "-------------"
print "FINAL RESULTS"
print "-------------"
print "cached .png file loaded at %.04f seconds per MB or %.04f seconds per file" % (image_reload_time/(pngsize*times_reload_same_image), image_reload_time/times_reload_same_image)
print "cached flat file loaded at %.04f seconds per MB or %.04f seconds per file" % (flat_reload_time/(flatsize*times_reload_same_image), flat_reload_time/times_reload_same_image)
print "flat files are %.1fx the size of the optimized .png" % (flatsize/pngsize)
print ".png files loaded at %.04f seconds per MB or %.04f seconds per file" % (image_load_time/(num_image_copies*pngsize), image_load_time/num_image_copies)
print "flat files loaded at %.04f seconds per MB or %.04f seconds per file" % (flat_load_time/(num_flat_copies*flatsize), flat_load_time/num_flat_copies)