Re: file access library

[Christian Reiniger <warewolf@mayn.de>]

Dan Kegel wrote:

>> 2) loading (mmapped or not) another big file will most likely purge the
>> disk cache, forcing most of the mmapped files to be re-read on the
>> next access
>This might not be a common event.

Well, you know, I like to be able to guarantee a certain behavior. That's
my main concern with mmapping - you never can be sure that your data is
really cached, it's just more or less likely.

>Also, in the case of a file which you *know* you won't
>need to read more than once (e.g. movies), there might be a
>way to tell the OS not to cache it.  This is a topic for

Hmm, yes. I won't be of much help with this since - as I already told you -
low level programming/system level stuff isn't one of my strengths...

>research... I have seen ways to hint about how much
>readahead to do, but don't know what Linux in particular offers here.

Well, if that's possible it would be a relatively good way.

>> 4) mmapping more data that can fit in the OS's disk cache at the same time
>> most likely (depending on the access patterns) wont give much speed gain or
>> even will slow things down - mainly depending of the size of the disk
>> cache and the amount of data accessed  in the same (short) period of time.
>It's ok to mmap a huge pakfile, as long as you don't access too much
>of it.  So the problem isn't mmapping, it's how you access it.

Yep, that's what I tried to say ;)

>Telling the OS to not cache certain very big accesses may be
>a win regardless of whether mmaping is being used or not.

Hmm, right. Didn't think about this... But it's a really good idea.

>> Ok, now what if we'd leave the file access system simply stdio-like -
>> copying data on access, no mmapping.
>> 
>> Then we add a system that caches data. We can give this system a
>> pointer-length pair to cache, we can instruct it to purge the entire cache
>> (e.g. on entering a new level that uses a different set of textures/meshes)
>> and we can instruct it to cache no more than n bytes (to prevent accidental
>> memory hogging).

>This is the traditional approach.  Nobody will fire you for going this
>route.

Ok, backing up ;)

I mainly said the "no mmapping" thing above because I wanted to concentrate
on the library-level caching system. 

I don't mind having mmapping support in the basic file access code - that's
a good thing if made properly and if presented as an option. But I also
like the library-level caching system because it allows for (1)
guaranteeing the caching of data and (2) caching-after-decoding, i.e.
making it possible to store data in a compressed form on the disk without
slowing things down significantly.

>I'll try to read up on Linux memory management.  I recall that
>this is a big topic for rewriting in the 2.3 kernel... perhaps
>we should pay attention to that discussion, and mention
>our requirements... if 2.0 or 2.2 don't have good enough
>memory management to make this mmap scheme practical, perhaps
>we can influence 2.3.

Hmm, yes. Who on this list is subscribed to linux-kernel?

Cu
	Christian
--

Windows 95 - Native Plug & Pray