Reverse engineering of Wipeout 3 SE

[aybe]

EDIT

Here is the suite of 4 utilities to unpack PBPs for WO3:


a GUI app that will process a folder of many PBPs
individual command-line tools to deal with PBP, CMP, TIM
(the UnpackCMP and UnpackTIM also work for Wipeout 1/XL/PC)


here: https://github.com/aybe/WXX-Rebirth/releases/tag/latest (check README.TXT)

There won't be anymore updates as I think they are mature enough : they successfully produced nearly 40K files (all WO3 versions). :clap:clap:clap

:g



Houston, we have track data !!!

http://img11.hostingpics.net/thumbs/mini_206349snap08102.png (http://www.hostingpics.net/viewer.php?id=206349snap08102.png)

After much struggling with those command-line utilities, here's finally some result !

There's still a lot to be done but this is motivating nonetheless.

:nod :nod :nod

[Xpand]

You're doing an amazing job man! Congrats! :)

[aybe]

Thanks :D

[tigrou]

Great job. I'd like to help but don't from know where to start.

Anyway: which track is this ? (I'm not a Wipeout3 fan at all, I mostly played WOXL and W1 (a little bit)).

I can't find it there : http://wipeout.wikia.com/wiki/Wipeout_3

Also why are textures like that (flat single color) ?

[Ace3000]

That's one of the Prototype tracks. You get them by completing certain things (I know it has to do with single race mode).

Also, the "flat colours" are part of the Prototype tracks art style. There are about 6 Prototype tracks in total, all of them have that minimalistic style except for a couple.

[aybe]

Great job. I'd like to help but don't from know where to start.

Anyway: which track is this ? (I'm not a Wipeout3 fan at all, I mostly played WOXL and W1 (a little bit)).

I can't find it there : http://wipeout.wikia.com/wiki/Wipeout_3

Also why are textures like that (flat single color) ?

I am going to publish a new tool and an Excel file prolly within 2-3 days, currently working on the spreadsheet to try identify what is what and so on ... the new tool is way better/faster, what used to take nearly 30 mins :evil now barely takes 2 mins -> the extraction of every PBP. I manually checked WO3SE and it pretty much recognized 99% of files, there are 39530 files BTW :g And the tool is a GUI, no more command line :turd, just choose input/output folders and watch it extract.

The plan:

find which PBP is which track
try attack the hintful ones (proto 2, 3)



Also I should get this 3d vertex picker up as I guess it'll be mandatory because while plot.ly (https://plot.ly) is great to quickly preview data, it doesn't show much infos.

[aybe]

Quick update

The new unpacker:
9458

A track's vertex data:
9457

Will post the unpacker soon !

[tigrou]

@aybe : its getting interesting, I hope you keep working on it !

Note : on wipeout central the W3 tracks drawings looks really precise.

http://s29.postimg.org/8mcc7gb93/latest_cb_20120521114641.gif

I would not be suprized somebody already found a way to parse track format...

[aybe]

1/
I didn't notice there were HQ drawings for the non-prototype tracks :clap

2/
For WO3, see my final release of the tools:
https://github.com/aybe/WXX-Rebirth/releases/tag/latest

They will also work for Wipeout 1/XL (for CMPs and TIMs), make sure to see the README.TXT first !

3/
I haven't really made any progress in WO3 beside fine-tuning and finishing the above tools, like I've seen you enhancing the WebGL experiment recently, I am focusing on it to get WXX-Rebirth on par with it first.

I've finished the loading of textured models and the surroundings such as packing textures:

9467

9468

9466

I am close to get the same LOD that there's in the experiment, the next big step in comparison to the experiment will be to object-ify scene objects so they can be used in a game.

4/
Regarding formats of Wipeout 3,

I'm pretty convinced the vertex data is the same since it's on PSX
Objects now are presents and there should be indices pointing to them, unpack a track's PBP and search for strings such as CUBE, you'll find them


When you unpack a PBP you will see that:

4 small files with lots of blanks
probably track-related data
track vertices
probably billboard stuff since textures are right after
track sections or something
track textures
2 groups of ship-related data (depends though)


Do not hesitate to share your findings or ask for help, multiple pairs of eyes are likely to help in deciphering all this mess.

5/
Basically once I reach about the same rendering as in the experiment I will take some time for WO3.

PS: in the WebGL experiment, do you know what the track faces and sections are representing ? thanks

:D

[Xpand]

Track faces are the triangles created by a set of three vertices. Their format is something like this:
You have a set of vertices:
vertex0
vertex1
vertex2
vertex3
...
vertexn

your faces are, in a generic way:
face(i): {index of vertex ,index of vertex,index of vertex}

example:
face1: {0,1,2}, uses vertex0, vertex1 and vertex2.

About the track sections I think it has to do with how to know what track section is a corner, what track section is the starting line, what track section is the pits, where the speed/item pads are, etc. probably for both texture placement and game logic

[miljoneir]

@aybe : its getting interesting, I hope you keep working on it !

Note : on wipeout central the W3 tracks drawings looks really precise.

I would not be suprized somebody already found a way to parse track format...

I guess these images came from the wipeout 3 'kleber' website.

[tigrou]

Hi,

@aybe : the following files are used in W1/WOXL :

- TRV : contains all track vertices (x, y, z 3D points)
- TRF : contains all track faces. basically an array of indexes that references the vertexes in TRV file. This allow same vertices to be shared between triangles (as XPand explained). All objects around the track (tunnels, mountains, bridges, trees, ...) are stored separately, in the SCENE.PRM file.
- TRS : track sections. this is a double linked list of 3D vertices that represent the track curve. in other words, a list of points that follow the track.

This is used for lot of things :
- physics : collision is performed only with triangles (track faces) in current section (where the player is), not the whole track.
- rendering : each section contains a list of 3D objects to display (precalculated before). So, the game only render triangles visible from player position (occlusion culling).
- game AI + autopilot (CPU ships knows which points to follow)
- checking who is 1st , 2nd (ranking)
- checking if player passed over a weapon / boost / repair tile, ...
- ...

[aybe]

Track faces are the triangles created by a set of three vertices. Their format is something like this:
You have a set of vertices:
vertex0
vertex1
vertex2
vertex3
...
vertexn

your faces are, in a generic way:
face(i): {index of vertex ,index of vertex,index of vertex}

example:
face1: {0,1,2}, uses vertex0, vertex1 and vertex2.

About the track sections I think it has to do with how to know what track section is a corner, what track section is the starting line, what track section is the pits, where the speed/item pads are, etc. probably for both texture placement and game logic

that totally makes sense, I was getting confused with PRMs but tracks are not PRMs, thanks :D

- - - Updated - - -


Hi,

@aybe : the following files are used in W1/WOXL :

- TRV : contains all track vertices (x, y, z 3D points)
- TRF : contains all track faces. basically an array of indexes that references the vertexes in TRV file. This allow same vertices to be shared between triangles (as XPand explained). All objects around the track (tunnels, mountains, bridges, trees, ...) are stored separately, in the SCENE.PRM file.
- TRS : track sections. this is a double linked list of 3D vertices that represent the track curve. in other words, a list of points that follow the track.

This is used for lot of things :
- physics : collision is performed only with triangles (track faces) in current section (where the player is), not the whole track.
- rendering : each section contains a list of 3D objects to display (precalculated before). So, the game only render triangles visible from player position (occlusion culling).
- game AI + autopilot (CPU ships knows which points to follow)
- checking who is 1st , 2nd (ranking)
- checking if player passed over a weapon / boost / repair tile, ...
- ...

Hi, thanks, that's really what I was looking for and is going to keep me busy for a while ! :D

I am close to get flat/textured models done and once it is I'll attack the tracks.

Have you tested the PBPunpacker and others I have posted ? (feedback appreciated)

[aybe]

I finally got textured objects working but I do have a strange effect, I can see through the model:

9471

Any ideas on what is causing this ?

Basically I use exactly the same code since those PRM structs are identical except that when it is a face, there is only one color; I am starting to wonder whether I mis-understand the logic of faces ... idk :frown:

thanks :)

EDIT : was my fault, SpriteBatch messes the GraphicsDevice, proper states should be restored before drawing :g

[tigrou]

I can see through the model

It's probably because back face culling is activated, and you are rendering triangle in the opposite order.

Try reverting the order you render triangle vertices (CW vs CCW). It can also depend of a flag inside the model.

[aybe]

It's probably because back face culling is activated, and you are rendering triangle in the opposite order.

Try reverting the order you render triangle vertices (CW vs CCW). It can also depend of a flag inside the model.

It's that SpriteBatch that messes everything, I was starting to wonder my code was not enough for faces but it works very well. Only the tracks are missing now and I'll have some great visuals then !

[Xpand]

That's pretty neat! I actually rebuilt the ships from scratch a few years ago, but it was through working my way through broken texture fragments and meshes that vincoof managed to extract from the graphics buffer, so some bits were off:
http://fc01.deviantart.net/fs71/f/2012/204/6/2/6286eb3b3fdecb0a1a16d115bf202225-d58ah2v.jpg

[aybe]

That's a really great render :clap :clap :clap

[aybe]

Hey,

just IDA-ed the MAC OS X of Wipeout and there are pretty cool things in it such as the name of subs:

9503

seems like they forgot to obfuscate it !

[tigrou]

@aybe : that is really cool.

C/C++ is always sort of "obfuscated" anyway (unlike C#/Java or more high level languages).
I think what happened is this : the game exe which is released for MAC OSX version is compiled in debug mode (by mistake probably or because MAC OS X C/C++ include that by default). This is probably what is used by IDA. or maybe RTTI is turned on (this is something similar to reflection in C#).

By looking at methods names it confirms one theory we discussed before : most objects around the track are animated done using custom, hardcoded methods (instead of being defined by data in PRM files).

EDIT : could you maybe post a full list of all methods (here or by PM?) thanks.

[aybe]

PM sent, also there used to be a patch 1.3 but I can't get hold of it. If by chance you find it, beware that StuffIt Expander for PC will not decompress it, I remember only being able to unzip from an old Mac ... If only we could get those structs ! IDA found the names but maybe it can find their signature...

This approach kind of alleviates the need to peek into files, it will probably fun to do though there might still be a few things we can figure from these unknown fields ... for the rest I'm pretty certain that there are things we won't be able to figure. IMO the things that are a must to find would be these animations such as the ones of the rescue ship but we'll see over time. The Unity approach is really cool but I guess at some point I should share that base code to write the viewer/picker I've talked about (I've simply copied classes and fixed them for Unity in like 2 hours)

BTW just found this : http://www.gamekult.com/blog/panda/309726/wipeout-2097-version-pc-hd-edition.html It pretends to run WOXL in HD and at the right speed, still need to test though ....

[tigrou]

@aybe : thanks

I checked quickly in the exe file you sent me (using that tool (http://split-code.com/strings2.html)), and it does not seems to contain any data structure information detail (unfortunately)

What we could do is try to find as much versions as possible of that game (eg : Saturn, Amiga) and check if it might contains some relevant stuff (eg : using utility I mention above).

[aybe]

Are you sure that an executable would contain literal names of struct members ? (cool website btw :D)

I couldn't look at the Saturn version at all since it's an 65Mb ELF file, there should be some unpacker for it on the web I guess...

PM-d you.

[tigrou]

Are you sure that an executable would contain literal names of struct members ? (cool website btw )


Yes, unless the executable is compressed in some form (data sections are unpacked in memory while running), but this is unlikely.
Take the exe file you send me and open it in notepad or use string2 tool. You will find all methods names and all parameters you extracted using IDA.

So in some structures you should find names stuff like "nextsection" "radius" "junction", "x", "y", "z", i expect to find these but did not see anything like this (and also nothing that would looks like structure members)

What is in MacOSX version is probably to debug (step by step) or to produce a stack trace if i something goes wrong at runtime (so you can easily find what goes wrong)

I think they forget to do a strip (not enabled by default): http://linux.about.com/library/cmd/blcmdl1_strip.htm
http://en.wikipedia.org/wiki/Symbol_table

[aybe]

I will give it a try !

Unfortunately I don't have the Mac anymore so I can hardly test, maybe through these VM emulation hacks ?

While they forgot to strip things out they probably haven't forgot to set Release mode ... that would have been too good to be true!

[tigrou]

What do you want to test using the Mac?

I have access to a recent IMac with OSX (not at home but close), so i can try some stuff as well if needed.

[aybe]

What you've said : debugging it with something like IDA.

If you want to give it a try, feel free ... I can also get hold of my old Mac but it's quite a pain to use : 800Mhz Ibook ...

Btw, didn't they drop the PPC emulation layer in today's macs ?

[miljoneir]

BTW just found this : http://www.gamekult.com/blog/panda/309726/wipeout-2097-version-pc-hd-edition.html It pretends to run WOXL in HD and at the right speed, still need to test though ....

They don't run it at full speed. The person just explains how to do it with other software (like MSI Afterburner OSD).
And sadly, all of the link are down. So no Wipeout @ 1080p for us :( The reg file can still be seen though, so it allows for installation of the game under a 64 bit os.
PS I'll post a comment in french down there and hope we'll get the downloads again.

[aybe]

yes just seen the links were broken, it'd be interesting to get hold of one EXE to understand how they are HD patched !

[twenty90seven]

I've re-enabled the link to the Amiga version

https://drive.google.com/folderview?id=0B2Kr-nVTNoAPcHFiQkhnbHZyYzg&usp=sharing

The assets are just the same as the PC version. But the Amiga version outputted at a higher resolution.

I used to have the Mac version but can't find it....I had it running on a Mac back in 2009. The Amiga version was better than the Mac...the Mac version had some effects missing - the blue light trail behind the ship did not look as good - it seemed to be missing some elements.

[aybe]

Thanks, thus far it hasn't yielded anything useful as the decompilation fails at some point, we should give it another try in the future !

[Sausehuhn]

Just so you know: You won’t be able to play WO2097 on any recent Mac, as it’s a legacy program that only runs on PowerMacs and these featuring Rosetta (http://en.wikipedia.org/wiki/Rosetta_(software)). OSX 10.6 was the last OS to feature Rosetta and newer Macs cannot run it at all – virtual machines being the only possible solution here.

[Cipher]

Up to a certain version of mac OSX you could still run in classic mode (was it tiger? Or panther? Or maybe even earlier?) which allowed you to run oldschool mac applications, if you don't have that, you can always dust of an old G3 or G4 (loved those machines) otherwise indeed, emulation is the way to go, and it's not easy, but definitely possible, SheepShaver is what i use, it's kind of complex to set up, but once it's running it works pretty much flawlessly ;)

Cipher

[miljoneir]

also there used to be a patch 1.3 but I can't get hold of it. If by chance you find it,

I have got it. I don't remember anything about the contents of the patch though (which files exactly), so I only send you the .app itself for now.

[aybe]

thanks !!! :)

EDIT : VirtualBox seems to emulate 10.6, going to try that,

[JABBERJAW]

The Mac version is no good. It skips the first 10 degrees of movement when using analog control, making it impossible to turn smoothly like the other versions

[aybe]

I don't really remember how it was ... anyway the emulation itself within VirtualBox seems doomed as the only usable version would be 10.6 but it simply panics ... seems like it's the unsupported processor (Core I7) driver doing this :paperbag

[Sausehuhn]

I had 1.6.8 running on an i7 Machine (from 2010, that is) and still have it on an old 2007 MacBook, if that helps. But I have no clue about the wizardry you guys do with those games ;)

[aybe]

It's for looking at it using a debugger to see whether we can understand/discover a bit more about the game's formats ... ultimately the most needed things are figured out but it'd be a bonus to unveil other things. The most interesting things are the scenery animations.

Just seen that patched iAtkos distribution thing but as you've laid out it's not really essential.

[UnleashSonic]

EDIT: Are the models for the ships easy enough to extricate from the game? I really want to 3D print some WipEout ships to test out my local 3D printing facilities but can't find any models of any of the ships from the WO games.

Sorry guys, hadn't discovered the other thread (http://www.wipeoutzone.com/forum/showthread.php?8897-Extracting-WipEout-3D-models-(large-images-allowed)) before posting that.

[miljoneir]

yes just seen the links were broken, it'd be interesting to get hold of one EXE to understand how they are HD patched !

Good news! That frnch dude responded and fixed the download links! All thanks too Panda (tm) from gamekult! Original link: http://www.gamekult.com/blog/panda/309726/wipeout-2097-version-pc-hd-edition.html
Direct link is here:
https://mon-partage.fr/f/zr8ziSdk/

[aybe]

Thanks !

[twenty90seven]

I'm finding the instructions on the website a bit hard to follow (even using Google translate)...has anyone installed the HD executable on a 64 bit system and can provide a walk through?

[aybe]

Follow this http://www.wipeoutzone.com/forum/showthread.php?9964-Running-Wipeout-2097-XL-in-Windows-8-1

Then replace the EXE,

The instructions says that the game will only recognize first CDROM so if your Daemon Tools is after real drive make sure to change its drive letter,

Also they say it only works in c:\wipeout but IIRC this is not true,

It will run very fast and those frame limiters didn't work in my case,

if all else fails you might want to try PCEM as now it has 3dfx emulation OR stick to EPSXE with very high resolution settings -> it will have fine rendering