[f-cpu] more about f-romfs

[Yann Guidon <whygee@f-cpu.org>]

here are some more thoughts and contextual stuff.
it's preliminary and moving, there's no code yet.

however i think it is a good answer to the people
who want a "BIOS" for F-CPU. First it is impossible
to do that on a platform where there is only a CPU
and then i understand that in the past years, the
PC BIOSes have increased in complexity to a point
where it's unmanageable. Having a small boot routine
with a romfs-like support is a cool solution.

i have to sleep now, after i read the comments that
i just got.

WHYGEE
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
f-romfs.txt
created mer mar 13 03:30:43 GMT 2002 by whygee@f-cpu.org

A discussion about a "file format", the associated algorithm
and the context of using a romfs-like system.

--------------------------------------------------------

BIOS/Monitor/extensions :

The F-CPU processor is designed to boot at address 0
after reset. It will fetch its instructions from a
non-volatile memory or an image of this data if the
system is simulated in VHDL or C. The boot code will detect and
initialize all the fundamental devices such as the memory
controllers. For example, the refresh rate will be tuned,
the bank size will be adjusted and the base address of the
private memory range will be set to an unused range
(if the system is comprised of several CPU with their own
private address ranges, but they must not overlap).

The boot memory also contains very low-level and primitive
routines for doing basic stuff, such as sending error
messages (to a virtual console that is mapped somewhere
in the CPU, probably the SR). When done, we have to
initialise a lot of other things which are independent
from the F-CPU project and we don't have the means
to work on the HDD controller, the keyboard driver,
the video framebuffer... You'll understand that we don't
want to mess with it, a single CPU is already complex enough.

We would like to let the user configure his system
and install only the necessary drivers, or customize it with
his driver for his prototype peripheral, etc.
But we don't have access to a file system because
the HDD is not yet setup ! this would bloat the code
or make it too complex. OTOH we would like to let the
user interact with the HW.

The solution is to use something that looks like
romfs : we have access to user-provided data and
code modules and we have to provide a few functions
that allow the modules to interact in a read-only way :
 - locate : return the physical address of the requested block
 - open   : returns a handle to the specified block
 - close  : close handle
 - read   : get a word
 - seek   : goto a specified offset in the "file"
 - exec   : specify a file and run it

since the additional modules are meant to
provide development or debug tools, it is useful
to use "directories" and "symbolic links".
This is done with the help of "flags", just like
a normal FS. we can add other flags : executable
(so a user won't attempt to execute raw data),
and compressed (in case we have some room for
the decompression tables and code, when the
Flash will be completely filled with tux
pictures etc...)

The Flash image will be built with a simple
  $ cp bareboot.bin newimage.bin
  $ cat modules.romfs >> newimage.bin
  $ installflash newimage.bin
(science fictious)

The bareboot.bin file is built from an assembly
langage source containing :
 - the startup code
 - RAM initialisation code
 - the minimum libraries : decompression, romfs utilities
   and some additional stuff as needed

The modules in modules.romfs are built from
an utility (kind of mkromfs) with a set of specified
files which are also assembled, using symbols extracted
from the startup/library code. The modules' code must
be position-independently coded because when executed,
the code (which can be decompacted) will be copied
and aligned in private/fast memory.

When all the minimal setup is performed (RAM and IRQ,
for example) then the boot code uses the romfs functions
to access the romfs part of the Flash, seeking a "startup"
binary which will then initialise the rest, using
internal scripts or facilities that the user can customize
at will. The file could be named "runfirst" for example.
If no romfs is found (starting at the end
of the boot block) or if the startup file is absent,
the CPU sends an error code to the virtual console
and enters an infinite loop or in shutdown state.

Granularity : F-CPU is a byte-addressing machine
but the architecture does not like unaligned instructions
or data access.
* Concerning the instructions, the code
blocks should be copied to the main memory and
the starting address must be aligned to the 256
bit boundary imposed by the cache lines. In fact,
code that is optimized for F-CPU aligns loop entries
to 32 byte boundaries and it would not be wise
to break this. Furthermore, the Flash memory is
a rather slow device and is likely to be uncacheable
(though its contents doesn't change, but you get
the idea...). The SDRAM memory has much more bandwidth
and can cope with prefetching so one can concentrate
on code efficiency.
* Concerning data : their access is often in a random
order (or no order at all) so the caching debate is
not the same. If all data accesses are done "in place"
with "read" function calls, all we have to do is to
align the data blocks on 64-bit boundaries (8 bytes).
This is the minimal and mandatory register size for
the whole F-CPU family so any compliant CPU can boot 
our system.

So the block alignment is 64 bits (8 bytes) and we
have to pack several data in the headers.


Scenario :
 1) the system boots a single kernel (Linux 4.8 for example)
 -> the kernel is compiled and the file is renamed "runfirst"
so the boot code starts the kernel directly, thus benefitting
from all the kernel's facilities such as device drivers ...

 2) multiple kernels :
 -> a multiboot binary is installed so the user can select
which kernel he wants today. However the user interface
is reduced to close to nothing : it has to first run the
primary user I/O setup code (a call to exec with a keyboard
driver then a video driver). When a kernel is selected,
the manager "exec"s the corresponding "file".

 3) development platform :
 -> it is possible to implement a bare shell (not as complex
as bash but useful anyway). This interface must load
the keyboard and screen modules as well as some romfs
extensions to provide directories, for example.
This is going to be useful in a simulated environment where
the kernel must be rebuilt, for example. An editor,
an assembler, a compiler, etc. are going to be provided
and a "flasher" can generate a new flash image.

 4) unsupported HW or old kernel :
imagine you install a new HDD or a sound peripheral device
which is not yet recognized by the kernel. or it's simply
in development phase, or simply there's no kernel at all.
Device dependent modules can be managed outside the bare
boot process, simply by providing an "exec" function. All the
decisions are left to the user.

This is nice because we have no time to write a monolithic
"BIOS" such as in the PC world, and it's so much more flexible.


File format (no name yet) :

 * the boot/setup/library code is assembled and its
size is rounded up to 8 bytes. The symbols are output
to a file which is #include'd by the other modules
so they can be assembled with the physical address of
the basic functions.

 * all the modules are assembled and the binaries
(and possibly, data files) are gathered by a "mkromfs"
and the result is concatenated at the end of the bare
boot image. This way it is possible change the modules
without touching the boot code.

 * a "directory" contains a header followed
by N data blocks. These blocks are copied
"as is" (unless compressed) and aligned to 8 bytes
(zero padded). The header contains :

 - size (4 bytes)
 - number of blocks/entries
 for each entry :
  - size of entry, size of the name,
      attributes (exec, compress, directory, link...)
    -> 64 bits
  - ASCIIZ file name
  - size of the block (in bytes)
  - index of the block (relative to the start of
      the romfs block, 8-byte aligned)

The variable file names are a problem. It could be possible
to limit it to 16 chars and use a fixed-size header structure
so we can also get rid of the entry size field.
ideas welcome. the "original" romfs format is a good idea
(the file name is merged with the variable sized block)
but i don't like to walk all around the ROM anyway
(if the Flash works with page modes, then we're in troubles
and contiguous accesses are better). It's not meant to
be a high performance system but who ever knows, i don't
like to make crap.


I have not yet addressed the need for a CRC of the
image. This is a heavy and slow process, but it might
be needed in some conditions anyway.

The locate, read and seek functions are pretty easy
to implement, even in asm. Managing internal structures
such as the read() buffers and handlers will also require
a sort of "malloc()". This becomes even more important
for exec or modules which need to expand data.

exec requires a call to open, read, close, malloc
and even a decompression code if the instructions
are compressed. However most flags are unused
and not likely to be useful before more development
is done. directories, links etc are not yet supported
and are not even necessary but the functionality is
available and reserved.

--------------------------------------------------------

Conclusion : this is slowly looking more like an "operating
system" but with some strong constraints like footprint
and (more difficult) the reduced I/O capabilities.
Unlike the PC there is no "default" I/O at a fixed address
and this makes the simulations inside a VHDL tool much
more difficult.

It remembers me that in the "early" computer ages, computers
ran with small "monitors" and no BIOS. Today and in the future,
the platforms evolve extremely fast and become so different
from each others that it is not possible to make any
assumptions as before.

One solution is to rely on the Linux or Hurd kernels because
most of the necessary I/O handling is already provided.
The problem is to provide enough basic I/O capabilities
so that the kernel can boot properly, display its messages,
get the minimum memory etc... This can be done with "modules"
that are also stored in a romfs-like system.

OTOH a read-only file system for a bare CPU is very easy
to implement. Only a reduced set of functions must be coded
so it's a good candidate for assembly coding, which in turn
gives a "live" example of how to write code for the F-CPU.