Serialization for wii

Hi !

I am making level editor on PC (wxwidgets around C for business functions) that can make binary serialized files with fwrite calls, of course the wii can't read these processor arch dependant files.

I've browsed the internet to search serialzation libs [en.wikipedia.org]

libtpl seems great, but it can't be compiled with devkitppc, because of POSIX functions dependances : mman.h, wrapped around with windows functions with mingw.
And I even can't compile c11n on my pc...

Maybe that the only way to get a 'wii gekko' binary serialised file is to do both ASCII / XML / ETC and fread/fwrite in the application, and on the wii read with ASCII then write with fwrite (turning the game in some kind of conversion application)?

So :
Is there C serialization library that can be compiled with devkitppc ?
Otherwise, how can I define my own portable binary serialization structure ?
I've seen games with .bin files (helium boy) how have they do that ?

Thanks ! ^^

Hi TheDrev,

I think probably the most common way of dealing with the kind of problem binary serialization would be useful for in C is to just load what you need from your file directly into malloc-ed memory and to use a system of pointer arithmetic to do the hard work. Similarly, you would write the binary data of your objects by providing a different kind of pointer (say, a byte array?) to their location in memory and treating it as such for your fwrite call.

You might find it helpful to include an offset table as a "header"-eque construct to provide your file with structure. It can be a little bit messy at times, but as long as you know what you're doing, it should be just as safe as, and much more efficient than, providing an explicit Serialize/Deserialize cycle.

Hope that helps!

- Myu0



Edited 2 time(s). Last edit at 04/05/2009 11:05AM by Myu0.

I am sure that you likely don't need full serialization. You just need a fileformat that isn't endian dependent. Just define the endianness for the format and make all programs use the same endiannes when working with the format. There is no rule saying that you can't operate with the wrong endian mode with some convertions.

thanks,
I understand what you mean, but it's too abtract now. Can you give an example ?

I'll do my best to come up with a pretty rough example. Given that I haven't actually tried this technique, and I'm not currently within easy access of my devkitpro libraries, I wouldn't copy this code wholesale! It's more to demonstrate the idea than to tell you how to do it.

Let's use a very basic scenario where each file corresponds to a particular struct instance. A level design struct might be drawn up something like this:

typedef struct {
 tileobject levelcontents[32]; // Defines an array of pre-rendered graphical tiles to build the level with
 u8 contentgrid[200][40]; // Defines a 200x40 grid of tile values
 u8 spawnpointx, spawnpointy; // Defines where the player starts
 u8 endpointx, endpointy; // Defines where the player ends
} levellayout;

The idea is that you can store the graphical and behavioural information within the level file as encapsulated by a tileobject struct, and then use a grid of integers as indices to the array of tileobjects at runtime. I won't go into too much details on the tileobject.

What you want to do as a level designer is for the run-time of your level design program to fill out the levelcontents array, the content grid and the spawn/endpoints, and to persist the contents of the resulting struct in a file of some sort. Now, once you've got this struct into a contiguous area of memory (having malloc-ed it previously) and set everything you want to set, writing it to a file could be done something like:

levellayout *level1;  // The pointer for our levellayout struct
void * mem;  // The pointer to get from malloc
if (( mem = malloc( sizeof(levellayout) ) ) == NULL) { // do the malloc - supposed to be whatever contiguous memory allocation function your OS uses
	printf("ERROR ALLOCATING mybuffer\n");
	exit;
}
level1 = (levellayout *) mem; // cast the pointer
{/* ... code to set the various struct values ... */}
int bytesize = sizeof(levellayout); // get the size in bytes of the struct
byte *level1bytes = (byte*) mem; // create a byte pointer to the malloc-ed memory
FILE *file;
file = fopen ("level1.bin","wt"); // open the file for writing
if (file != NULL) {
	for (int i = 0; i < bytesize; i++) {  // step through the array and write the bytes.
		fprint("%c",level1bytes);
	}
	fclose (file);
}

Now that you've got this file, which just contains a byte array, you can include it with your app. In order to make use of it, you read it into a memalign-ed section of your Wii's memory when you need it (which'll be pretty large for the given example). Then, you just recast the pointer back to a levellayout. Your wii function might look something like:

void * mem;  // The pointer to get from memalign
mem = memalign(32, sizeof(levellayout));
byte * level1bytes = (byte *) mem;
{/* ... Code to read the bytes from the file to the level1bytes pointer; probably would use a buffer over a for loop similar to before, and too lazy to code myself right now! >:3 */ }
levellayout *level1 = (levellayout *) mem;
... // use the level1 data in memory as necessary

There might be a more eloquent solution, but in terms of providing basic binary serialization, that's a pretty simple way to do it.

Hope it helps!

- Myu0



Edited 2 time(s). Last edit at 04/05/2009 10:43PM by Myu0.

Sure, you can just do a blind faith byte dumping serialization. But it's trusting the player to not load faulty data. And it does not work very well if you have pointers in the data. And if you are using c++, you are likely using some of the standard templates. And those may (and extremely likely are) use pointers.

And why are you using fprint for a binary file? Use fwrite for such jobs.

Oh, and it still doesn't do anything about endianness.

Like I said, general idea, not precise implementation! Couple of questions though:

Quote
henke37
Oh, and it still doesn't do anything about endianness.

Is that a problem? I'd have figured you just wrap what you write to your file around a simple translation function to write according to whatever endian mode you want to run in (which I think is Big on the Wii by default, so yes, you would need to swap the order of the bits in each byte from most architectures, but that's not terribly complex to do).

Quote

Use fwrite for such jobs.

My bad!

Quote

But it's trusting the player to not load faulty data.

True, but all you'd need to add to fix that would be a signature and a few checksum tests to make sure the file hasn't been tampered with. If players muck about with the output they get from a level designer, failure to load is quite reasonable behaviour.

Quote

And it does not work very well if you have pointers in the data.

It's a matter of good memory management, really. Don't write pointers in your files unless you know they're pointing in run-time to where they should be, but if you do then feel free!

Quote

And if you are using c++, you are likely using some of the standard templates.

If you wanted to use C++, you could make all this redundant anyway by providing a Serializable base class and having the necessary objects implement their own serialization. C doesn't have that luxury.

- Myu0



Edited 3 time(s). Last edit at 04/06/2009 12:09PM by Myu0.

Let me put it this way, when you write pointers for a loader that is not even run on the same cpu arch, you can safely assume that not a single pointer will be the same. I consider a pointer invalid once the program has been shutdown.

Luckily, there is an easy trick for serializing pointers. You remap them to a value that will not change between serialization and unserializarion. One such value is the position in an array in the same serialization data. Another way to do it is to save the address of each object before serialization and save the pointers as they are. Then you can just do a pointer remapping pass at unserialization.

Ok, seems hacky, in the good meaning :)
I'll check this later, see if can do something like that. For the moment I'll use the mini xml lib and use xml. As I said in my first post, I can also load the data with XML and serialize theme on the wii and then erase xml for the release.

Also, I have just compiled gdbm (gnu hach library, like a data base ) with devkitpro, I will test it on the wii this week, if it's work I'll put libs on the wiki.



Edited 1 time(s). Last edit at 04/06/2009 06:37PM by TheDrev.