About the .PBP file format

[aybe]

UPDATE 4

Take a look at notes in update 3 then,

https://github.com/aybe/WXX-Rebirth/releases/tag/PBPv2


TIMs detection
bug fixes



At this point you are getting pretty much the same picture as in the attached Excel file, what is not an image is game data; things are clearer.

There won't be anything new until quite some time now, things needs to be reverse-engineered.

:D :D :D



Update 3


it has been simplified, accepts one file and an output directory
all bugs are fixed AFAIK
better interface/reporting with colors
shipped with an Unpack.bat to make your life easier


The older version was trying to be smart and failed somehow, now there are two tools that each do a very specific task and both do work well. I've ditched the multiple input files feature as this was limiting the output options and was cumbersome, with the current version you can write your own batch file and keep control of things as you wish.

Please disregard all previous versions.

There will be a last update (or unless bugs are found), the detection of TIM files; for the time being they are easy to spot : every file that has two sequence of numbers (e.g. TRACKM0A.PBP.00011.00000) is a TIM. You can easily group them within Explorer by sorting by modified date :

http://img15.hostingpics.net/thumbs/mini_809121snap0809.png (http://www.hostingpics.net/viewer.php?id=809121snap0809.png)

(in this example I've already renamed them to TIMs but they would pop in the exact same order)

grab it here (https://github.com/aybe/WXX-Rebirth/releases/tag/PBP)

PS1: to preview TIM files in Explorer, grab SageThumbs (http://sourceforge.net/projects/sagethumbs/)
PS2: your AV might suspect these EXEs, there's nothing I can do about that and you have to allow them, also you can always check the code with JustDecompile (http://www.telerik.com/products/decompiler.aspx)

Waiting for your feedback now !

:D



UPDATE 2

Two new tools (UnpackCMPEx and UnpackPBPEx (https://github.com/aybe/WXX-Rebirth/releases/tag/UnpackCMPEx_UnpackPBPEx))

searches for CMP files and unpacks them
unpacks a PBP (simplified version of the first tool, better suited for batch operations)



So far, it successfully unpacks a PBP and the CMPs inside it, no false-positive AFAIK :+

http://img11.hostingpics.net/thumbs/mini_491824snap0808.png (http://www.hostingpics.net/viewer.php?id=491824snap0808.png)

See USAGE.TXT for an example !

PS1: TIMs are not yet renamed to .TIM as explained in USAGE.TXT, working on another tool for that.
PS2: you can use SageThumbs (http://sourceforge.net/projects/sagethumbs/) to get a preview of TIMs within Explorer

:P


UPDATE 1

Here is (finally) the first version of the unpacker : https://github.com/aybe/WXX-Rebirth/releases :P

I guess it works properly since it extracted 10990 items out of 197 files (every PBP file from every version: WO3E, WO3J, WO3SE, WO3U, WO3UDEMO)

NOTES:


full instructions on how to use it are on the README.TXT
archives of TIM images (aka CMPs) are not decompressed by the utility -> the UnpackCMP utility is on its way for this.
in the web link above there is also an Excel file, it contains a couple of informations already figured from the unique PBP file in the Wipeout 3 US Demo Disc (highly recommended since most files will harbor this structure)
the 'unofficial' preferred version of Wipeout 3 to study is the SE version, it is the most recent, the most featured and surprisingly there are less files


Let me know if you encounter any problem !



Never say never, all things have a beginning but if we don't start we have zero chance to succeed :g

I took a quick look at the format and here are my findings:

http://img11.hostingpics.net/thumbs/mini_275356snap0794.png

(http://www.hostingpics.net/viewer.php?id=275356snap0794.png) http://img11.hostingpics.net/thumbs/mini_721806snap0795.png (http://www.hostingpics.net/viewer.php?id=721806snap0795.png)

Header (in UINT32 units)

at offset 80 it is the number of entries in the container
at offset 88 starts the 12 bytes struct of each entry
1st value is absolute offset in file
2nd value is unknown
3rd is the length of the entry
i've checked on multiple PBP and they fit perfectly with their length


We can already get the individual parts of these PBPs and try to get to next step :hyper

IMO these files are certainly compressed but not necessarily encrypted, there used to be some entropy detection software that could help but can't remember the name actually ...

EDIT

Just wrote a tiny unpacker, it swallowed every PBP :hyper out of WO3SE from \WIPEOUTC and \WIPEOUT3; there is always between 58 and 60 files in each PBP. With time and patience we'll decipher them :cowboy

EDIT 2

I don't know where he got these infos (http://www.wipeoutzone.com/forum/showthread.php?1207-Modifying-Wipeout-3-SE&p=14652#post14652)from but they're hintful.

EDIT 3

Some progress ! :hyper

There are VAG files in these containers: (ship thruster sound)

http://img11.hostingpics.net/thumbs/mini_296345snap0796.png (http://www.hostingpics.net/viewer.php?id=296345snap0796.png)

There are also a bunch of files which I suspect are textures since there are many entries in their header have a 0x40 value (64 pixels), not .TIM files though. We should prioritize in searching for files from the official formats in the SDK, if not then it could simply be a form of RLE encoding.

There are also files that looks like polygon data but it's a bit premature IMO :

http://img11.hostingpics.net/thumbs/mini_544004snap0797.png (http://www.hostingpics.net/viewer.php?id=544004snap0797.png)

The left-most pixel rarely changes and could indicate the polygon type as it can be seen in no$psx debugger.


Also many of these files are duplicate, basing myself on studying a group of files from the link posted in edit 2.

2 be continued

- - - Updated - - -

EDIT 5

Houston, we have TIMs from PBPs !

http://img15.hostingpics.net/thumbs/mini_794119snap0798.png (http://www.hostingpics.net/viewer.php?id=794119snap0798.png)


:hyper :hyper :hyper

- - - Updated - - -

EDIT 6

http://img15.hostingpics.net/thumbs/mini_305880snap0799.png (http://www.hostingpics.net/viewer.php?id=305880snap0799.png)

- - - Updated - - -

EDIT 7

Houston, we're about to crash on vertex data !

http://img15.hostingpics.net/thumbs/mini_161572snap0800.png (http://www.hostingpics.net/viewer.php?id=161572snap0800.png)

:hyper :hyper :hyper

basically, files with no headers are simply LZ77 streams

EDIT 8


Large 2 groups on bottom : ship/team data and I guess textures
Bigger files seems to be vertex data as they have some kind of stride

http://img11.hostingpics.net/thumbs/mini_339471snap0801.png (http://www.hostingpics.net/viewer.php?id=339471snap0801.png)

[aybe]

Last update for today :cowboy

The content of the 8 last files :

http://img11.hostingpics.net/thumbs/mini_420578snap0802.png (http://www.hostingpics.net/viewer.php?id=420578snap0802.png)

[Xpand]

Wow, great work, man! :)

Just out of curiosity, what programs did you use? I only have an hex editor and gcc... xD Makes things so much tougher.

[tigrou]

Yep, great work. If you keep going that way we will have a Wipeout3 model/track viewer soon!

[aybe]

Wow, great work, man! :)

Just out of curiosity, what programs did you use? I only have an hex editor and gcc... xD Makes things so much tougher.

Only an hex editor but it's quite featured as you can see above (coloring, etc) and the infos from wipeout.js; the rest I wrote it in C#

- - - Updated - - -


Yep, great work. If you keep going that way we will have a Wipeout3 model/track viewer soon!

this is going to take time, I think I'll post my results somewhere so people can try by themselves.

- - - Updated - - -

To the admins:

Can we post a decompressor of these files at the forum, or not ?

[tigrou]

Only an hex editor but it's quite featured as you can see above (coloring, etc) and the infos from wipeout.js; the rest I wrote it in C#
To the admins:
Can we post a decompressor of these files at the forum, or not ?

What about creating a github repository ?

[aybe]

Yes, probably within the next days, there are a couple of things I would like to sort before:

- references because I've just been coding against Monogame and WPF and those utilities do drag lots of unneeded DLLs
- get my renderer on par with Dominic's one (textured objects + Wipeout 2), shouldn't be too hard.
- clean up the thing a bit, write a README etc ...
(those are imminent, easy tasks)

I'm done with the PBP/CMP/LZ77 unpackers, still trying to figure out whether I can properly detect LZ77 streams so one wouldn't have to bother to manually check whether it is encoded or not (to get a 1-click PBP unpack)

other things of interests (later)
- add viewers to the GUI app. basically you select a file and you can see 3d objects etc ...
- implement a viewer with triangle selection, we'd instantly know what object is what data hence help us understand the thing better
- begin the coding related to the game :g

regarding previous talks it seems that the way to go would be to have another name than Wipeout, been thinking about WXX-Rebirth (like http://www.dxx-rebirth.com/), what do you think, sounds cool isn't it ? :D

- - - Updated - - -

Just 'secured' the thing : https://github.com/aybe/WXX-Rebirth :hyper

I'll post the WO3 unpackers there ASAP ...

[aybe]

About to post an unpacker, which I guess properly understands the format now (its behavior matches my manual checking of every file, while previously it was not). The final step is to properly detect CMP streams and I think we'll be done with the outer shell. Will push command-line tools hopefully by tonight :cowboy

[tigrou]

@aybe :

one question : is Wipeout 3 simply using a container (PBP) and then the inside is files similar to WO1/WOXL ? Or is the inside of PBP something totally different from what we have seen before ?
I'm not talking about common PSX files (like TIM textures) but rather wipeout specific track formats (like TRS/TRV/TRF, ...)

[aybe]

Some things differs definitely, some not.

PS: just found out that US versions are PBN instead, but they;re working as well (they have different content though)

- I guess the vertices are the same and mostly for polys, though we'll only know once we progressed.
(from what I've seen with that little app that renders as bitmap, i.e. PCM sounds are instantly recognisable and those that I suspect being the above twos do look like WO1/2)

- there is an object system with names (cubeXX, goteki, etc ...) my initial thoughts were to attack the format by studying the 2nd prototype track as it's very simple and contains cubes, these are small structs that once we figure one of them we should progress quite a bit. I think I've figured where are the indices but it's a bit puzzling.

- i do not suspect other things to differ that much since as we're on a PS1 and as you can see in no$psx, the rendering possibilities are known, there aren't 100s of possible combinations (tri, quad, face, vert, etc ...)

This in conjunction with a vertices renderer should significantly help, basically if you render an array of vertices as spheres and look at them from a certain point of view you'd know very quickly if's vertex data or not; e.g. the 2nd prototype track will be of help because there are orphan cubes in the circuit, a significant hint. (will test that theory with WOXL data first as we know what to expect)

for the rest you'll discover that by yourself :g

|
|
V

UnpackPBP is done and works well, still would like to throw all the pb(n/p)s to test it, writing a little doc, a few other things then I'll put on GitHub. :hyper

It's been quite time consuming to get from a pseudo-format to a well understood (i guess by now) format. Maybe tonight or tomorrow, actually time runs pretty fast when you do something interesting !

[aybe]

Unpacker released at the top of this page :nod

[aybe]

The detection and unpacking of CMP files inside PBPs is mostly done:

http://img15.hostingpics.net/thumbs/mini_196672snap0807.png (http://www.hostingpics.net/viewer.php?id=196672snap0807.png)

- - - Updated - - -

New update ! (see 1st post) :nod

[Sausehuhn]

This thread feels like watching a detective solving his case. Keeps me checking it every day :P

[aybe]

I think I'm done by now, just shipped (http://www.wipeoutzone.com/forum/showthread.php?9953-About-the-PBP-file-format) what I hope will be the latest update; now comes the real reverse-engineering work !

[aybe]

Another and last update ! no more until a major milestone now ...

[Amaroq Dricaldari]

I hate to be a buzzkill, but the Github links are broken.

[aybe]

Sorry ! I'm not receiving any notifications for some reason ...

The repository has gone private, and they're gone anyway !

But I have a backup here : https://1drv.ms/u/s!AqAtR29m3cqGtR6XE-PtUrbLNpg7

[Amaroq Dricaldari]

Thanks.