Re: PenguinFile

[Christian Reiniger <warewolf@mayn.de>]

To the list: this started as a private discussion, so don't wonder if you
don't understand everything ;)

Bjarke Hammersholt Roune wrote:

[ppfOpen () time]
><Well, partly typo, right. It should be 1e-5 seconds ;)
[...]
><The hdd's seek time, the time it takes to move the reading/writing "head"
><to the position of the file. That's a time in the single digit millisecond
><area.
><
>Ahh, yes, what I mean was, though, what is the head positioning TIME?
>What range will it lie in on a medium-size computer today?

Well, in the single-digit millisecond area ;)

><>I was working under the misconseption that an open file in a pak file
><>took up resources. This is apparently not so, and therefore you are
><>naturally corrrect.  
><
><Well, it *does* take up resources - for each opened file a ppFILE structure
><has to be allocated. But that's nothing serious (~ 40 Bytes).
><
>40 bytes? It seems to me that that seems just a bit excessive, but then
>I haven't gone thorugh the code to see what it's used for, so perhaps I

Well, some positions, sizes etc - that easily sums up. But 40 bytes really
aren't that much. 

><If you e.g. read a wav file (say some speech bit of a monster), 30k big,
><you will spend about 10 microseconds in ppfOpen (), then wait about 8
><milliseconds until the hdd head is positioned properly and another some
><milliseconds until the data is read. That means the ppfOpen () part is
><about 0.1% of the entire process.
><
>10 micro seconds? I thought it was 1 micro second? (10e-5)

10e-5 = 10 * 1e-6 = 10 * 1 microsecond

><Now if the data was already in cache at the time the situation would be
><entirely different, but if you don't know that you'll need a given file you
><usually don't have it in cache either.
><
>How long does it take then? (to read around 30 kb)

Well, the seek time is 0 and the reading is reduced to a RAM-to-RAM copy.
That means all expensive operations are gone and the overall time will also
be in the microseconds area.

><>You could instruct the pak file generater to generate headers which
><>specified the path or URL of files in the headers rather than their
><>numbers, except on the rare occasion that it was building a release build.
><>It could be an option.
><
><Then ppfOpen () needs to be *really* overloaded - and the API would be
><unusable from plain C...
><
>Not really, you would only need a single overload; one that took a
>number, and one that took a string. The string version, I take it, is

The thing is that C does not support function overloading at all.

><>Though, most files ARE really hardcoded into the game, if you look at

><Hardcoding files in the game is bad practice IMHO. It's perhaps OK for
>that little oh-lets-write-another-pong-clone-tonight game, but for bigger
>games it's really bad.
><
>Depends on what you mean by "hardcoded". They wouldn't be hardcoded in
>that they coudln't change, so they could be updated.

Only by recompiling the game. With "hardcoded" I mean that the names of all
(or most) of the used files are specified in the code, not in some data
file.
A good handling would e.g. be to have a fixed main configuration file
("mygamerc") that contains the name of the levels' main map files, which in
turn contain the names of the mesh/texture/soundeffect/... files they use.
So you can exchange all game data by simply copying some files around.

><Occasionally I download a bunch of levels/maps for Quake or Halflife, copy
><them to my game dir and thest them (-> the simplicity I talked about) and
><after a while when I need some more space on my hdd I just remove the bad
><ones. That's one of those scenarion I'm thinking about.
><
>Yes, I see how that behavior is nice (you don't have to tell the game
>where the maps is, you just need to tell it to play them).
>
>However, you couldn't do that with this feature, atleast not the way
>you've sketched out to me.

Of course the game needs to support this a bit. The overloading-for-updates
is just one of the scenarios. Shoddy argumentation from my side ;). The
important thing is that these things require that the file names are *not*
compiled into the game.

><>Giving the PenguinFile client a way of updating pak files is a good
><>idea, I think, regardless of wheter it is possible to overload mounted pak

><That's relatively easy to implement (2 or 3 function calls) either in the
><game or a seperate utility. I don't think an extra API function for this is
><needed.
>
>It is a feature that just about every larger game will need, and while,
>no, it's not a major hassle to implement, I don't think it's as trivial to
>do as you say it is. These are the things you would need to do to update a

Yes it is ;)
(read on before you reply to this)

>1- build a new pak file, combining the old pak file with the update one,
>comparing file dates along the way.
>2-delete the old pak file.
>3-rename the new pak file to the name of the old.
>
>to build the name pak file, you would need to do this:
>
>1-create a complete mirror of the directory structure and file names of
>every file in the old pak file, aswell as any any new files in the update
>pak file.
>2-pass this structure to some function that will then build the new,
>updated pak file 
>
>Doing number 1 is not an extremely hard job, though it's not very
>interesting to implement, and it's also something that I imagine can create
>a few memory leaks. Not any work here to speak of.
>
>As for doing number 2, I wasn't able to find any PenguinFile API that
>wrote pak files. The closest thing I could find was ppf_WriteInPak, which
>should only work for format 1 pak files. (it isn't implemented either :)
>
>I don't know if there is suchs an API, but I couldn't find it (I didn't

Of course, because it's only in my head yet. I haven't described it in the
docs yet because I'm not 100% sure about how the functions will look like
and behave. Just 95%.
Here's a quick rundown on the main new API functions:

ppfCreatePak (const char *Name, <all the header parameters>);
will create a new, blank PakFile. For filling it it needs to be mounted
somewhere.
This is the most uncertain one of the functions - I'm not sure if
the create-mount-fill-umount thing is the best way and the "header
parameters" thing will most likely replaced by a struct holding all those
info (passing >= 8 parameters to a function is evil and the number of
paramers will slightly differ between the Pak formats).

ppfCopy (const char *source, const char *dest, int flags);
will copy the file or dir at source to dest - recursively if needed (or
specified).

That means updating a pak will require roughly that code:

while (there_is_another_update)
{
	ppfMount (TheUpdatePak, TheCommonMountpoint, ppfMF_override_files);
}

<prepare header info>

ppfCreatePak (NewPakName, HeaderIfo);
ppfMount (NewPakName, TmpMountPoint, ppfMF_write);

ppfCopy (TheCommonMountpoint, TmpMountpoint, ppf??_recursive);

ppfUmount (everything);

>The point is that doing this from the game is not that much work for the
>game programmers. It shouldn't take even a very unskilled programmer more
>than an hour, at most, to do it. However, as virtually everyone will be
>implementing this behavior, I don't see why a ppfUpdate() feature should
>not be included.

See, it's just unneccessary ;)

>I would go for having both an utility and an api function. This way,
>programmers can call the API directly, but they (and everybody else who
>migth want to update a pak file) can also do it from ... well, the linux
>command line interface (I got a bit of vocabulary to learn, it seems :)

"shell" or "cmdline" for short ;)

><>A) I don't actually have linux. I've made up my mind to get it one of
><
><You could do a Win32 version of PFile.
><
>Besides the POSIX file handling, is there any other incompatibilities in
>PenguinFile towards WIN32 I should know about?

Not yet. The memory mapping stuff will be however.

><Well, one of the nice things with Linux is that it already comes with one of
><the best compilers available - gcc
><Plus a multitude of other development tools - debuggers, build and
><configuration systems, doc generators, excellent editors etc. So don't
><worry about this issue ;)
><
>There's a compiler in the kernel?!? I take it you mean that most
>distributions come with a compiler... ;)

'fcourse ;)

>This gcc, does it do things like templates? What about member function
>templates? (one of the reasons I'm geeting VC 6.0 is that VC 5.0 doesn't do
>member function templates). How old is this gcc? Does it have bugs?

Yes (except for a very minor exception), yes, 0.2 - 25 years ;), propably,
but I haven't found one yet.

About the age: The first version was IIRC written in the seventies, and the
newest version in about 2 months old.

>Isn't there any integrated development environments? IE, do you then
>have to write make files and invoke the compiler, builder and debugger from
>the command line?

There are some IDEs, but most of them are in a relatively early development
stage. the LGDC pages (http://sunsite.auc.dk/linuxgames/) list some of them.
But even if a real superduper-mega-killer-IDE was available, I'd still
prefer the cmdline. IDEs always put all kinds of junk in your source dir,
makefiles etc. And I haven't found a single IDE with a decent text editor
yet ;)

><Well, the bad thing is that I'm in the middle of a greater change in PFile
><so there is no stable code to play around with. But you could nevertheless
><already have a look at the code. The internal directory handling code for
><example is more or less in its final state. Are you familiar with cvs or
><should I prepare a tarball (or zip) of the sources?
><
>I wouldn't exactly say I'm familiar with cvs, but I did manage to get my
>hands on the code. I wasn't easy though! first I got a program called
>WinCVS, but after half an hour, I simply had to conclude that it just
>didn't work, or, more probably, I couldn't get it to work (wouldn't
>connect). Then I got the "real" command-line driven thing and sorted it out
>in about half an hour. Had a little problems with it getting my password
>form the keybord...

*grin*

>I tried to get the thing to compile in VC++, but, as you might imagine,
>that was impossible. If you could get me the POSIX file handling headers, I
>think I could get it to working though. It wouldn't build, but then that's
>ok for now.

Descriptions coming via privmail.

>The make file also doesn't work for VC++. You'd really think that that
>kind of thing would work the same for all compilers on all platforms, but
>well, it doesn't :)  It stops at the very first line complaining that there
>shouldn't be a + sign in the line.

Well, VC++ make is primitive compared to GNU make ;)
And our makefiles also make use of autoconf, so converting them is even
harder. You'll have to create your own makefiles/project files from scratch.

>Btw, why are you using POSIX-specific functions? Why not just do
>everything through the standard library?

Because the standard library has no functions for directory reading.

>Something just occured to me: if all the basic functionality has to be C
>in PenguinPlay, how on earth are you going to get that event system going?
>It's templated to it's very core...

We won't ;)
We have a simpler procedural event system for the lower levels
(cvsdir/src/PenguinEvent/). The one you're talking about is only for the
higher levels.

>pps. Is the mailinglist usually so quiet, or am I just not subscribed
>properly to it? This "emzlm" program did send me a message saying that I
>was subscribed, but there hasn't been a single message for two days.

It is rather quiet, but not *this* much. Did you respond to the mail you
got from ezmlm? Unless you do that you won't get anything.


	Christian
--

Drive A: not responding...Formatting C: instead