Re: file access library

[s369625@student.uq.edu.au]

On Tue, 15 Sep 1998, you wrote:

>> 3) mmapped files have to be in the OS's disk cache as long as they are
>> being accessed to give a speed gain. On the contrary, if they're not in
>> cache when the data is accessed the system is slower than a streamed,
>> copying approach (stdio-like)
>Right, if the files get dropped from the cache when there's
>really enough room to keep them in memory, it would be awful.
This is something we have to worry about with streamed access too. In other
words, even if the mmaped pages get dropped out, we aren't really losing
anything compared to streamed access.

 

>> cache and the amount of data accessed  in the same (short) period of time.
>It's ok to mmap a huge pakfile, as long as you don't access too much
>of it.  So the problem isn't mmapping, it's how you access it.
>Telling the OS to not cache certain very big accesses may be
>a win regardless of whether mmaping is being used or not.
I should assume the OS can figure things out for itself.  If I understand linux
correctly, it will not read it into the cache untill:
	1) It has some spare memory AND it is idle time
	2) You actually access the file.
Well, that's obvious, but...



> 
>> Ok, now what if we'd leave the file access system simply stdio-like -
>> copying data on access, no mmapping.
>> 
>> Then we add a system that caches data. We can give this system a
>> pointer-length pair to cache, we can instruct it to purge the entire cache
>> (e.g. on entering a new level that uses a different set of textures/meshes)
>> and we can instruct it to cache no more than n bytes (to prevent accidental
>> memory hogging).
In a way this caching is no better than mmap.  Remember, our worry is that in
times of short memory linux will drop mmaped pages.  If we do our own caching,
then linux will simply swap the cache out.  If we trust linux to swap memory
inteligently, we can trust it to handle mmap well too.  So the only difference
become that a swap partition would be faster than a filesystem, I'd ignore that.

Better to leave the kernel to do this sort of thing, that's what it is for.  At
the end of the day, any usage pattern that could be considered random
access is at least as quick as anything else you could do.


>> files by supplying a
>> ppFOpenMM (const char *FileName, void **Pointer, int *length);
>> function. This code reads the requested file via the file access system,
>> decodes it into a directly usable form (e.g. jpeg -> raw RGBA), hands
One thing, File access does not care about format conversions!  If PPlay needs
some support for efficiently caching converted formats etc.  that opens all sort
of design issue which are not in the scope of file access.


>
>> >Mmap access can be used for absolutely everything, even streaming
>> >video, if you like.
If you do it carefully.