In xoreos proper, the most visible change is in Knights of the Old Republic: seedhartha has put a lot of work into making the tutorial partially playable! The player character can collect their belongings, put on their gear, let Trask open the first door, open the second door and engage in mock combat.
You can see it in action in this video here:
Under the hood, the code for Knights of the Old Republic and Knights of the Old Republic II was fundamentally restructured to build off a common base, so a lot of the underlying concepts in the first game are also available in the second, but are not as visible. You can, however, skip the prologue in Knights of the Old Republic II to watch your player character awaken in the Peragus mining facility and trigger a few early interactions.
Watch it in this video here:
The only other game where something has happened visibly is Jade Empire: you can now run through a preliminary character generation and listen to area background music.
User-invisible changes include:
xoreos and xoreos-tools now require a C++11-capable compiler, in an attempt to modernize a bit. This should hopefully not be a huge problem. Phaethon already required a C++14-capable compiler the last release; this has not changed.
This leads us to the news in xoreos-tools: the xoreos-tools package has gained nine new tools, most of them thanks to Nostritius. These tools are: unobb, untws, rim, keybif, tws, fixnwn2xml, xml2gff, fev2xml and ncsdecomp.
The first two new tools, unobb and untws, are new archive unpackers. unobb extracts “obb” files found in Aspyr’s Android and iOS ports of the BioWare games, which can be either plain ZIP files or, more interesting, a virtual filesystem. untws extracts save files from The Witcher.
The next three tools, rim, keybif and tws are archive packers. rim is the counterpart to unrim, creating RIM archives. keybif is the counterpart to unkeybif, creating KEY/BIF archives (and lzma-compressed KEY/BZF archives found in Aspyr’s mobile ports). However, V1.1 files for The Witcher are not yet supported. And lastly, tws, is the counterpart to the new untws tool, creating save files for The Witcher.
Next up, fixnwn2xml takes the non-standard XML files in Neverwinter Nights 2 and turns them into valid, standards-compliant XML files. This tool is thanks to rjshae, based on asr1’s work, reworking it to fit better into our stream classes.
xml2gff is the counterpart to gff2xml, taking an XML file and turning it back into a GFF. Only GFF3 (GFF V3.2/V3.3) are supported for now, so neither Sonic Chronicles: The Dark Brotherhood nor the Dragon Age games (which use GFF V4.0/V4.1) are supported at the moment.
Another work-in-progress tool is fev2xml, which reads the FMOD event file format FEV and creates a human-readable XML file. Only the FEV1 version is supported and then only a fraction of its features.
Likewise, ncsdecomp is the start of an NWScript bytecode decompiler, built on the foundations of our NWScript bytecode disassembler. It’s highly experimental and we give no guarantees that it works correctly at all.
In addition to these new tools, there are some new minor features and bugfixes:
As for Phaethon, the 0.0.6 version of Phaethon is mostly a “maintenance release”, keeping the foundation in sync with xoreos. There are no major new features. There are, however, a number of smaller bug fixes. Also, Phaethon can now open ERF V2.1 files and display swizzled Xbox SBM images. Additionally, BIF files can be inspected directly, even when no corresponding KEY file can be found (this does mean, though, that the filenames are missing).
To sum it all up: xoreos inched a bit further along, gained more initial implementations useful later on and xoreos-tools hopefully increased its usefulness for modders and spelunkers. Still a lot more to do, a long way to go. But we are, after all, in for the long ride. :)
Sources and binaries for Windows, GNU/Linux and macOS are attached to the GitHub release, here for xoreos, here for xoreos-tools and here for Phaethon.
Additionally, the repository and the source tarballs contain PKGBUILD files in dists/arch/, debian build directories in dists/debian/ and spec files in dists/fedora, which can be used to build Arch Linux, Debian/Ubuntu and Fedora packages, respectively. Alternatively, the PKGBUILD files are also in Arch Linux’s AUR (here for xoreos, here for xoreos-tools, here for Phaethon), and we have a Gentoo overlay.
And as always, we’re looking for more developers to join us in our efforts to reimplement those 3D BioWare RPGs. If you would like to help, please feel free to contact us. :)
]]>I already wrote a combination introspective and release post back in July, which gave some updates since the previous Not-Thanksgiving post in 2016, so let’s see what happened since then:
Quite a lot, right? :)
Of course, there’s still things in progress, among them:
xoreos wouldn’t be what it is without the help of a lot of great people, for whom I am thankful.
Hopefully, the future will bring even more movement into this little project. If you would like to help with that, then please, feel free to contact us! :)
]]>A lot has happened in the last two years. A busy real life unfortunately made me miss the yearly “Not-Thanksgiving” progress report last year…so this here will be a kind of combination release post and progress report. Buckle up, this will be a long one. If you’re only interested in the short release notes, move on to the GitHub release pages linked above and again at the bottom of the post. Cool? Cool.
So, continuing from the last progress report in 2016, what exactly has happened?
Back then, I said I was working on unit tests for xoreos, planning to make them public in December. I did that, and then merged them in February 2017. Since then, several more have been added and bugs were found (and prevented) with them. I consider them a huge success and a boon to the codebase. However, the coverage is nowhere near 100%. Pull requests adding new unit tests to existing code are always welcome ;).
Soon after, I took another stab at one variant of the texture format in Jade Empire, used for lightmaps. I had previously tried to figure it out, but failed. But with the added knowledge of the textures in the Xbox version of Knights of the Old Republic, which are swizzled [*], I realized that the lightmaps in Jade Empire are also swizzled. They’re even swizzled in the same way, so I could just reuse the deswizzling code, and voilà, now that texture variant is correctly read.
([*] Swizzling changes how the pixels of the image are stored. Usually, pixels are stored linearly, from left to right, from top to bottom. This is not necessarily ideal for GPUs, though, because GPUs like it when things that are displayed near each other are also stored near each other (spacial locality). And while the pixels left and right of a pixel are near each other, the pixels above and below are not. Therefore, swizzled textures store pixels in a sort of zig-zag order. The Z-order curve is a popular scheme for that.)
Next up, Farmboy0 and Nostritius both worked on the GUI systems in Jade Empire and the two Knights of the Old Republic games. Jade Empire now has a partially working main menu (with dressupgeekout submitting pull requests to make the “Exit” button work and enabling background music), and the Knights of the Old Republic games gained character creators and partial in-game menus.
Speaking of Knights of the Old Republic, these two games are progressing quite a bit, thanks to seedhartha. They now have a fully working animation system, PC movement (with walkmesh evaluation based on Supermanu’s work) and even initial dialogues. In fact, the first game already shows a semblance of the first few minutes of tutorial on the Endar Spire, with your bunk mate telling you grab your gear from a chest and equip it.
For the Dragon Age games, Nostritius implemented an ActionScript interpreter. Together with a renderer for Adobe Flash vector graphics (still work in progress), this will be a reimplementation of Scaleform GFx. Scaleform GFx is used for the user interface, like menus and quickbars, in Dragon Age: Origins and Dragon Age II. Note that the vector renderer is not yet in the public xoreos codebase and that the screenshot below is more of a proof of concept.
On the xoreos-tools side of things, I added two new tools, ssf2xml and xml2ssf. ssf2xml takes a sound set file (SSF), as used by the two Neverwinter Nights and the two Knights of the Old Republic games and converts it into a human-readable XML file. xml2ssf is the counterpart, taking an XML file and converting it back into a game-readable SSF file.
Nostritius then added a third new tools, erf, an ERF packer. It is the counterpart to the already existing unerf tool. Where unerf extracts files from ERF archives, erf takes files and creates a new ERF archive. Unlike the unerf tool, erf currently only supports version V1.0 of the ERF format, as used by Neverwinter Nights, the two Knights of the Old Republic games. Jade Empire and The Witcher.
Also on xoreos-tools, cosmo-ray unified the parameter handling. This massively cuts down on code duplication in the tools and removes a potential source of mistakes.
Moving on to the third package, Phaethon. Phaethon is a graphical resource explorer, able to look into the archives found in the BioWare Aurora games and display several different types of resources. Previous xoreos releases didn’t mention it, but it has existed for several years already now. Of course, it’s still far from finished and could use more contributors.
In the past, Phaethon used the wxWidgets libraries for the GUI. But I was never really happy with it; I never managed to get widget placement to work consistently across platforms (or even just across different window managers on Linux). I always hoped to rewrite it in Qt some day. Therefore, I was really quite happy that michaelpm54 did that work for me, rewriting Phaethon to use Qt5.
I’m now far more comfortable providing Phaethon release packages along with xoreos and xoreos-tools. However, due to us using Verdigris to compile Qt5 applications without moc, Phaethon now requires a C++14-capable compiler. My local cross-compilers, the ones I use to create the binary release packages for Windows and macOS (and the chroot I use to build the Linux packages), didn’t have that. Consequently, I had to update my cross-compilers and chroot.
I did that, and now I can create working binary packages for Phaethon on these operating systems. And this also opens up a path to let C++11 features into the xoreos codebases in the future, something which I had been thinking about for a while now (currently, apart from Verdigris in Phaethon, all of the xoreos codebases are C++03-compliant).
On the other hand, also on the packages front, I dropped the openSUSE Build Service (OBS), the service that builds binary packages for a number of Linux distributions. I simply do not have enough time to keep the build specifications there up-to-date. Especially since every time a new distribution release is added, it requires me to invest several days worth of work to get them to build. Anybody willing to take over that task and officially administrate the xoreos OBS specs, please contact us.
Last but not least, there were also, of course, a lot of code quality improvements. Many of which were triggered by the introduction of new compiler warnings flags, new compiler instrumentations and static analyzers. I’m really grateful for all the options I have for checking the xoreos codebase, grateful for all the amount of work done by the gcc and clang/llvm people, and all the services offering free access to FLOSS projects.
And that’s about it for the past two years. Many improvements, but also still many things left to do. There’s several things cooking, but I’d rather not jinx them ;). In either case, I’ll try getting out the next release relatively sooner.
Sources and binaries for Windows, GNU/Linux and macOS are attached to the GitHub release, here for xoreos, here for xoreos-tools and here for Phaethon.
Additionally, the repository and the source tarballs contain PKGBUILD files in dists/arch/, debian build directories in dists/debian/ and spec files in dists/fedora, which can be used to build Arch Linux, Debian/Ubuntu and Fedora packages, respectively. Alternatively, the PKGBUILD files are also in Arch Linux’s AUR (here for xoreos, here for xoreos-tools, here for Phaethon, and we have a Gentoo overlay.
And as always, we’re looking for more developers to join us in our efforts to reimplement those 3D BioWare RPGs. If you would like to help, please feel free to contact us. :)
]]>A lot of things happened with xoreos this past year, albeit most of them hidden and “under the hood”:
Additionally, there are several tasks currently being worked on, among them:
From my side of things, my current plan is to make my unit tests branch public some time in December. I’ll write a small announcement here about it then. A new release of xoreos, 0.0.5, should follow early next year.
As always, this all wouldn’t have been possible without a lot of people. For them I am thankful.
I am also thankful for all the people who take the time to explain things to others, people who write interesting, useful or needed articles, and people who provide mentoring and help. Relatedly: a week ago, Stephanie Hurlburt published an article with engineers who are willing to mentor or answer programming/engineering questions. I for one think that’s a really great idea. Please take a look at that article.
And now, let’s see what the next year has in store for us. If you, however, found all this terribly interesting and would like to help with our little project, then please, feel free to contact us! :)
]]>In this release, Neverwinter Nights now shows speech bubbles for conversation one-liners, as used for cutscenes, bark strings and short NPC dialogues. Additionally, the premium modules BioWare sold for Neverwinter Nights, including the three that come with the Diamond Edition, can now be properly loaded and started.
An oversight in the handling of the texture fonts used in Neverwinter Nights and the two Knights of the Old Republic games has been fixed. This oversight broke rendering of certain characters, most prominently of those used in eastern European languages and the “smart” single quotation mark that’s used instead of an apostrophe in some strings found in the French versions.
For xoreos-tools, there’s two new tools: fixpremiumgff and ncsdis.
The first tool, fixpremiumgff, can restore the deliberately broken GFF files found in the BioWare premium modules for Neverwinter Nights. The resulting GFF files can then be edited as normal.
The second tool, ncsdis, is a disassembler for the stack-based bytecode of BioWare’s NWScript scripting language. It supports the scripts of all games targeted by xoreos and can disassemble them into a full assembly listing. It can also produce a control flow graph in the DOT description language, which can then be plotted into an image by using the dot tools from the GraphViz suite.
Moreover, this release includes a lot of user-invisible code documentation and quality fixes, in both xoreos and xoreos-tools.
Binaries for Windows, GNU/Linux and Mac OS X are attached to the GitHub release, here for xoreos and here for xoreos-tools. Additionally, packages for various GNU/Linux distributions can be found on the OpenSuSE Build Service (here for xoreos, here for xoreos-tools) and in Arch Linux’s AUR (here for xoreos, here for xoreos-tools).
Alternatively, the repository and the source tarballs contain PKGBUILD files in dists/arch/ and a debian build directory in dists/debian/, which can be used to build Arch Linux and Debian/Ubuntu packages, respectively.
And as always, we’re looking for more developers to join us in our efforts to reimplement those 3D BioWare RPGs. If you would like to help, please feel free to contact us. :)
]]>First of all, what is NWScript? NWScript is the scripting language and system BioWare introduced with Neverwinter Nights and used, with improvements and changes, throughout the Aurora-based games. Specifically, you can find NWScript driving the high-level game logic of Neverwinter Nights, Neverwinter Nights 2, Knights of the Old Republic, Knights of the Old Republic II, Jade Empire, The Witcher (in combination with Lua), Dragon Age: Origins and Dragon Age II. This is nearly every single game xoreos targets. The only exception is the Nintendo DS game, Sonic Chronicles: The Dark Brotherhood, which doesn’t seem to use any scripts at all.
NWScript is written in a C-like language and saved with the .nss extension. A compiler then translates it into a stack-based bytecode with the .ncs extension, which is what the game executes. That is similar to how ActionScript in Flash videos works, and how Java, Lua and other scripting languages can operate.
Like C, NWScript is a strongly typed language: each variable has one definite type. Among the available types are “int” (32-bit signed integer), “float” (32-bit IEEE floating point), “string” (a textual ASCII string) and “object” (an object in the game world, like an NPC or a chest). Moreover, there are several engine types, like “event” and “effect”, though which of these are available depends on the game. There are also structs, but in the compiled bytecode, they vanish and are replaced by a collection of loose variables. Likewise, the “vector” type is treated as three single float variables. A special type is the “action” type, a script state (or functor) that’s stored separately.
Additionally, Dragon Age: Origins adds a “resource” type (which, in the bytecode, can be used interchangeably with the “string” type) and dynamic arrays. Dragon Age II in turn adds the concept of a reference to a variable to help performance in several use-cases. For these new features, these two games each add two new bytecode opcodes, something not done for any of the other post-Neverwinter Nights games.
To get and modify the state of the game world, like searching for objects and moving them, and for complex mathematical operations like trigonometry functions, the NWScript bytecode can call so-called engine functions. These are functions that are provided by the game itself; about 850 per game, with some overlap. They’re declared in the nwscript.nss file (nwscriptdefn.nss for The Witcher and script.ldf for the Dragon Age games) of each game.
The original Neverwinter Nights toolset came with a compiler, but a part of the modding community, the OpenKnights Consortium, created their own, free software compiler, nwnnsscomp. Unfortunately, it has a few disadvantages. For example, it always needs the nwscript.nss file and it also only handles Neverwinter Nights. And while there has been several variations that have been extended to handle newer games, many of these are only available as Windows binaries. As far as I’m aware, there never has been a variation that handles Dragon Age: Origins or Dragon Age II. Also, since the code hasn’t been touched for 10 years, it’s difficult to compile now, and it doesn’t work when compiled for 64-bit. For what it’s worth, I mirrored the old OpenKnights NWNTools, with a few changes, to GitHub here.
This nwnnsscomp also has a disassembly mode, which can convert the compiled bytecode into somewhat human-readable assembly. This is pretty useful! I wanted my own disassembler in xoreos-tools.
The steps to disassemble NWScript bytecode are the following:
1) Read instructions
Read the .ncs file, instruction for instruction. An instruction consists of an opcode (like ADD
for addition), the argument types (which are taken from or pushed onto the stack) and any direct arguments. For example, an addition that operates on two integers would be known as ADDII
. The instructions are stored in a list, one after the other.
2) Link instructions
Each instruction may have a follower, the instruction that follows naturally. For most instructions, this is the instruction next in the list. But certain branching instruction, jumps and subroutine calls, also have jump destinations that may be taken.
3) Create blocks
Group the instructions into blocks. A block is a sequence of instructions that follow each other, with two constraints: a jump into a block can only land at the beginning of a block and a jump out of a block can only happen at the end of the block.
4) Create subroutines
Group the blocks into subroutines. A subroutine is a sequence of blocks that gets jumped to by a special opcode, JSR
, and returns to the place from where it has been called with RETN
. (In many programming languages, for example C, these are also called functions, but we’re calling them subroutines so that they’re not being confused with engine functions. Subroutine is also often the usual term in assembly dialects.)
5) Link subroutines
Record where a subroutine calls another and link the caller and callee, so that a call graph could be created easily. Likewise, the instructions that start and end the subroutine are also separately recorded.
6) Identify “special” subroutines
There are three special subroutines that we can identify:
SAVEBP
opcode.main
or StartingConditional
function visible in the script source. If a _global() subroutine exists, this is the last subroutine called by _global(). Otherwise, it’s the last (and only) subroutine called by _start().7) Analyze the stack
This step goes through the whole script and evaluates each instruction for how it manipulates the stack. Since stack elements are explicitly typed, and instructions that create new stack elements know which type they create (either explicitly, or implicitly by copying an already typed element), both the size of the stack and the type of all elements can be figured out. At the end, each instruction will know how the stack looks before its execution. And for each subroutine, we then know its signature: how many parameters it takes, what their types are and what the subroutine returns.
However, this step only works if we know which game this script is from, because we need to know the signatures of the engine functions. And, unfortunately, this step fails when the script subroutines call each other recursively. The stack of a recursing script can’t be analyzed like this.
8) Analyze the control flow
So far, the script disassembly consists of blocks that jump to each other, with no further meaning attached. To extract this deeper meaning, we analyze the control flow for higher-level control structures: do-while loops, while loops, if and if-else blocks, together with break and continue statements and early subroutine returns. Each block gets a list of designators that show if and how it contributes to such a control structure.
9) Create output
Finally, we create an output. This can be one of three:
As an example, here’s a script from Neverwinter Nights 2: this is the original source, and this is the full listing output, the assembly-only output and the control flow graph.
During this work, I have found a few interesting little bugs in the original BioWare NWScript compiler. For example, this little script, hf_c_skill_value.nss in Neverwinter Nights 2 and its disassembly:
1 2 3 4 5 6 |
|
1 2 3 4 5 6 7 8 9 |
|
Specifically, the int nSkill = GetSkillRank(nSkill, oPC);
line is compiled wrong. First, an instruction with the opcode RSADDI
is generated, which creates a new integer variable on the stack, for nSkill
. Then, the arguments for GetSkillRank
are pushed onto the stack, and GetSkillRank
is called using the ACTION instruction.
Unfortunately, as soon as the compiler creates the stack space for the local variable nSkill
, it associates nSkill
with this local variable. So when it’s time to push the parameter nSkill
for GetSkillRank
on the stack, the parameter to the outer StartingConditional
subroutine has already been overruled, and the CPTOPSP
points to the new local variable.
This renders the nSkill
parameter unused and useless, and GetSkillRank
is potentially called with an uninitialized value.
For another example, have a look at this script from Neverwinter Nights:
1 2 3 4 5 6 7 8 9 10 |
|
It’s meant to check whether the player character is a druid. Since you can multi-class in Neverwinter Nights, it checks whether the character is a druid for the first class, for the second class and then the third class. If any of these return true, iResult
will be set to true. To achieve this, a boolean disjunction (“or” operation) is used. As is customary in C-like languages, the boolean disjunction in NWScript is supposed to support short-circuiting: if the first part is already true, the second (and third) checks aren’t even called.
Let’s see how the disassembly graph looks like:
The first EQII
is the first comparison, and then the block in loc_00000057 is supposed to do the short-circuiting. It duplicates the top-most stack element with a CPTOPSP -4 4
before bypassing the second comparison and jumping to the LOGORII
that does the boolean disjunction. Unfortunately, instead of directly jumping to loc_00000080 with a JMP
, a JZ
was generated instead. And since the top-most stack element was already duplicated and checked with the previous JZ
, we know that the true-edge is never taken. It is a dead edge.
This has an interesting consequence. The short-circuiting for the boolean disjunction is broken: all parts are always evaluated before the results are or’ed together. In practice, this doesn’t matter much. It makes the code a bit slower, and any side effects will always happen.
Additionally, if the true edge were ever taken, the stack would be in a broken state. Unlike JMP
, JZ
consumes a stack element, and so the LOGORII
would be missing one of its arguments. Because this is not possible, it doesn’t matter for execution, but my stack analysis dies there. To combat this problem, I added an extra disassembly step after the block generation, the detection of these dead edges. To keep it simple, I only do some simple pattern matching, which is enough for most scripts. There are a few cases where it fails, though.
This bug is present in the original scripts coming with Neverwinter Nights, Knights of the Old Republic and Knights of the Old Republic II, but has already been fixed by the release of Neverwinter Nights 2.
This bug is also not present in the OpenKnights compiler:
As you can see, the branch instruction in loc_00000057 is a JMP
, as it should be.
So, to recap, xoreos-tools now has a tool that can disassemble NWScript bytecode, similar to the disassemble mode of the OpenKnights nwnnsscomp, with these added features:
If you’re interested, the source is available here. Binaries will come with the next release of xoreos and xoreos-tools.
There is, however, still a lot left to do there:
Unlike nwnnsscomp, xoreos-tools is still missing a compiler as well. This is something that would be very nice to have indeed. An assembler, which can take the disassembly output and create a working .ncs file out of it would probably be a useful first step in that direction.
If you would like to help and take up any of these tasks, or any other task from our TODO list, please contact us! :)
]]>First of all, what happened in the last year?
This is all old news, more or less already discussed in previous blog posts. However, since then, I added yet another new tool to the xoreos-tools package: ncsdis. It’s a disassembler for NCS files, the stack-based compiled bytecode of the C-like NWScript, BioWare’s scripting language used throughout their Aurora-based games.
It basically replaces the disassembler within the old OpenKnightsN WScript compiler, with various added benefits. I’ll write a bit more about this tool in the near future, so for now I’ll just leave you with an example assembly listing it can produce, as well as a control flow graph it can create (with the help of Graphviz). As you can see, it already groups the instruction by blocks and subroutines. It performs a static analysis of the stack (to figure out subroutine parameters and return types) and it also analyzes the control flow to detect assorted control structures (loops, if/else). I plan to grow it into a full-fledged NWScript decompiler.
Additionally, I also added support for BioWare’s Neverwinter Nights premium modules, like Kingmaker, to xoreos.
On the documentation side of things,
Phew! This is again a bigger list than I had anticipated. This wouldn’t have been possible without these people, for whom I am thankful:
Now that I have these mushy feelings out of my system, here’s hoping for another great year! :)
And like always, if you want to join our effort, please don’t hesitate to contact us!
]]>This release features a working script system for all targeted games, with game scripts being fired for the start of a campaign or module, when entering and leaving areas, and when clicking on in-game object. The singular exception is the Nintendo DS game Sonic Chronicles: The Dark Brotherhood, which doesn’t seem to feature any scripts at all.
The vast majority of engine functions, the functions that are called by the scripts and that do the actual work of tracking and changing the game state, are still missing, though. Per game there are about 850 functions (with some overlap) that need to be implemented. We currently have about 90, per game, of these written and working within xoreos. Moreover, many of the functions still missing depend on features not yet implemented.
Apart from the script system changes, 0.0.3 also comes with support for reflective environment mapping in Neverwinter Nights and the two Knights of the Old Republic games. The “metallic” armor and area parts that were rendered transparent in xoreos are now properly reflective. This can be seen, for example, in the Sith troopers in Knights of the Old Republic, in various plate armor worn by NPC in Neverwinter Nights, as well as the metallic floors on the planet of Taris and the icy wastes of Cania. For Neverwinter Nights, xoreos now also correctly smoothes the vertex normals of (binary) models, so that the metallic effect is not broken by sharp polygon edges.
On the xoreos-tools side of things, there’s now a new xml2tlk tool that can convert XML files created by the tlk2xml tool back into a talk table TLK file. Please note that, at the moment, only non-GFF’d TLK files can be written, as used by the two Neverwinter Nights games, the two Knights of the Old Republic games, Jade Empire and The Witcher. TLK files as used by Sonic Chronicles: The Dark Brotherhood and the two Dragon Age games can not be written (they can, however, be read with the tlk2xml tool).
Additionally, the convert2da tool gained the ability to write binary 2DA files, as used by the two Knights of the Old Republic games; and xoreostex2tga can now correctly read TPC cube maps.
Binaries for Windows, GNU/Linux and Mac OS X are attached to the GitHub release, here for xoreos and here for xoreos-tools. Additionally, packages for various GNU/Linux distributions can be found on the OpenSuSE Build Service (here for xoreos, here for xoreos-tools) and in Arch Linux’s AUR (here for xoreos, here for xoreos-tools).
Alternatively, the repository and the source tarballs contain PKGBUILD files in dists/arch/ and a debian build directory in dists/debian/, which can be used to build Arch Linux and Debian/Ubuntu packages, respectively.
And as always, we’re looking for more developers to join us in our efforts to reimplement those 3D BioWare RPGs. If you would like to help, please feel free to contact us. :)
]]>While xoreos is still far from being useful to end-users, all targeted games work insofar as that they at least show basic in-game areas. You can start the game, xoreos loads the game resources, loads a campaign or module, and then shows an area of the game. This accurately demonstrates what the xoreos project wants to accomplish.
Within the in-game area, you can fly around in a “spectator” mode, using the common first-person WASD control scheme. Moving the mouse while holding down the middle mouse button rotates the camera. With Ctrl+D, a debug console drops down, allowing for general resource dumping and the loading of different areas, modules and/or campaigns.
A few games, specifically Neverwinter Nights and Knights of the Old Republic, also show a main menu, although the latter’s is not as extensive yet. The former also shows a few in-game menu elements.
Additionally, Neverwinter Nights also has a script system hooked up, and preliminary dialogue support. This means that clicking on an NPC opens up its conversation dialog, and some of the script commands will be executed. For example, the door in the first area of the original campaign’s prelude opens after speaking to Bim and telling him that no tutorial is necessary. However, triggering the tutorial leads to the scripts looping endlessly, because the necessary game functions are not implemented yet.
Further gameplay is still missing. At the moment, none of the other games have a script system.
The current graphics are very basic: only flat-shaded, textured meshes are shown. No lighting, shadows or shaders of any kind are currently available.
Please note that xoreos is still missing a GUI and needs to be started from the command line.
The accompanying xoreos-tools package includes command line tools that can be used to inspect the games’ resource files and, as such, are meant primarily for developers.
Binaries for Windows, GNU/Linux and Mac OS X are attached to the GitHub release, here for xoreos and here for xoreos-tools. Additionally, packages for various GNU/Linux distributions can be found on the OpenSuSE Build Service (here for xoreos, here for xoreos-tools) and in Arch Linux’s AUR (here for xoreos, here for xoreos-tools).
Alternatively, the repository and the source tarballs contain PKGBUILD files in dists/arch/ and a debian build directory in dists/debian/, which can be used to build Arch Linux and Debian/Ubuntu packages, respectively.
And as always, we’re looking for more developers to join us in our efforts to reimplement those 3D BioWare RPGs. If you would like to help, please feel free to contact us. :)
]]>Lucky for me, the Dragon Age model format is reasonably well documented in the Dragon Age toolset wiki. tazpn even created standalone model viewers for Dragon Age: Origins and Dragon Age II, and released them with sources under the terms of the 3-clause BSD license. :)
And since the model format is based on GFF4, missing pieces of information are relatively easy to decipher too. So I quickly had a loader capable of reading the skeleton whipped up for both Dragon Age: Origins and Dragon Age II (since they are nearly identical in format).
With a bit of fiddling, the meshes were there too. There’s two types of meshes within the models: static meshes, directly hanging at one specific bone, and dynamic meshes that include weights for several bones for each vertex. Similar to models in Sonic Chronicles, this would deform the mesh according to those weights when the bones are animated. Unlike Sonic Chronicles, the default vertex positions of those meshes create a valid, unanimated pose. This means I could just completely ignore the bone weights for now, and load the meshes as if they were static. In the future, a vertex shader would combine those weights with the bone position to create the fully animatable model meshes.
Only thing missing now were the textures. For that, I needed to read the MAO (material object) files, which contains the material file (MAT), various textures (diffuse, lightmap, etc.) and a number of optional parameters. The material file in turn contains several different “semantics”, which is basically the name of a shader and how to map the MAO values onto the shader input. The original game takes all these, looks for the most fitting semantic in the material file (depending on number of parameters, graphics card capability and user settings), and then tells the graphics card which shader to use to render the mesh.
Now, since we don’t actually support any shaders yet (and we can’t use the game’s Direct3D shaders directly anyway), we simple read the MAO (which can be either in GFF4 or XML format), take the diffuse texture, and apply it to the mesh directly.
With the models done, I turned to reading the Dragon Age: Origins campaign files. A campaign, that is either the default single player campaign (which is defined in a CIF file), or a DLC package (with both a CIF file and a manifest.xml) that doesn’t extend another campaign (those would be add-ons).
There’s several caveats involved here:
First of all, most of the DLC packages are encrypted. The original game queries a BioWare server for the decryption key, asking whether its a legitimate copy. While the encryption method is known (Blowfish in ECB mode), xoreos does not include any of the keys. So the only campaigns apart from the main one loadable right now are the unencrypted ones, namely Dragon Age: Awakening, and any custom ones you might have downloaded (including the PC Gamer promo DLC A Tale of Orzammar).
Then, we don’t load any add-ons. So no Shale or Feastday Gifts, even if they weren’t encrypted (which they are). It’s not like xoreos could do anything with them yet anyway.
Finally, we have no way to install .dazip packages yet, so those need to be installed using the original game for now, or manually extracted and put in the right places. In the future, something that install them would be nice. Or maybe we could support loading of packed .dazip files, but that could be slow.
In either case, I implemented the loading of standalone campaign files.
Next up were areas (ARE) and environment layouts (ARL) with room definitions (RML). The ARE contains dynamic room information, like what music to play, and the placeables and creatures (more of those later). The ARL defines what rooms are in the area (as well as pathing information, weather, fog, etc.), each of them being a RML file with models. They are all, again, GFF4 files, making them nice and easy to understand.
There was one problem, though. The orientations of the models were given in quaternions, and as I said in the blog post about my The Witcher progress, a combination the automatic world rotation xoreos does, and our Model class wanting Euler angles instead leads to them not being correctly evaluated for whole models.
I was getting sick of that not being correct. I bit the bullet and removed the world rotation (which meant I had to rejigger the placement code in all engines, as well as the camera system, which was especially painful in Sonic Chronicles). And then I changed the Model class to take axis-angle rotations instead; those can be more easily calculated from quaternions, and can still be directly fed into OpenGL.
As a result, the area room models in Dragon Age: Origins were correctly oriented. And the placeable models in The Witcher as well.
You might notice that the ground mesh in outdoor areas looks very blurry and low-res. That’s because the original game doesn’t specify a single texture for those, but instead combines several textures together in a shader. We don’t support that yet, so instead we apply the replacement texture of the lowest LOD which is normally used for meshes that are far away.
On to the placeables, the objects within areas. They are defined within a list in the ARE file (giving position, orientation, name, etc.), each with a template. The template is a UTP file, a GFF3, that contains common properties for all instances of this placeable. This includes an appearance, which is an index into a GDA (a GFF’d 2DA, a two-dimensional table), which specifies, among other things, the model to use.
So far, so usual for BioWare games.
One difference, though. In the Dragon Age games, the GDA files do not stand alone. Instead, each is a combination of potentially several GDA files with the same prefix (defined in m2da.gda). This is used for DLCs, which then can simply add rows to a GDA, instead of overwriting the whole file. Consequentially, the appearance index is not a direct row number, but corresponds to a value in the “ID” column.
A bit fiddly, but still relatively easy to implement.
The creatures were more difficult. There’s several types of creatures: type S (simple) are just a single model; type H (head) are split into a body model and several models for the head (base, eyes, hair, beard); type W (welded) are similar to H, but already include weapons in the body model; and “P” (player-type) creatures are segmented into head (with base, eyes, hair, beard), chest, hands (gloves) and feet (boots).
Moreover, creatures of type P also switch model parts depending on the equipped items. So armor changes the chest model, gloves and boots change the hands/feet models and a helmet replaces the hair. Which models to use depends on several factors, and includes look-ups in several different GDA files, as well as UTC (creature template) and UTI (items) files.
Another problem is the tinting. The original game uses a shader to tint hair, skin and armor parts custom, user-selectable colors. To do that, their textures just contain intensity values in two color channels, while the two other channels are used as a bump map and something else (which I’m not sure yet). If we just apply the texture to those body parts, they are suddenly mostly transparent. To work around that for now, we manually modify each of those textures to remove the transparency. That leaves the weird coloring, but you can at least see all the body parts then.
I then applied all this to Dragon Age II. Just a few minor changes to the resource loading was necessary, and nearly everything worked out of the box.
Only the P-type creatures needed a bit more work, since how the body part models are constructed changed.
Similar to Sonic Chronicles, Dragon Age II is also missing many of the GDA headers; they’re only stored as CRC hashes. With a dictionary attack, I did manage to crack about half of them, but that still leaves about 450 unknown. Something to watch out for in the future.
I also investigated how music works in the two games. Dragon Age: Origins uses FMOD, and Dragon Age II uses Wwise. Both work similarily: the area specifies an event group, and the scripts then tell the library to play a specific event list from that group at certain times. The library does the rest, evaluating the events in the event list (which range from “play sound X”, over “set volume to Y”, to “add Z% reverb”). And while I do have adequately licensed code to read the sounds from both libraries’ soundbanks, figuring out the events is a massive undertaking. And we don’t have a script system for the Dragon Age games in place anyway, so this is nothing that can be done right now.
So… All games xoreos cares about now show areas. What’s next, then?
Well, first of all, I’d like to do some cleanup of the engines code. Sync them up, make them more similar to each other. Right now, many things are done slightly different in each engine, because the games changes something around and the old concept suddenly didn’t fit anymore. If possible, I’d like to unify the concepts again.
There’s also a few potential portability issues I want to investigate. For example, I read that using fopen() on Windows with filenames containing non-ASCII characters won’t work at all. Instead, I’ll probably have to change xoreos’ File stream class to use Boost’s fstreams, and convert our UTF-8 strings to UTF-16 on file open. I hope that’s something I can test with Wine, otherwise I’ll have to bug somebody with access to a real Windows.
After those things have been cleared, I’d like to prepare for our very first release. I plan to include both xoreos and xoreos-tools, with sources (of course) and pre-compiled binaries for GNU/Linux, Mac OS X (>= 10.5) and Windows, each for both x86 and x86_64. I have cross-compilers for those, and they all should work. Yes, xoreos is still not really useful for end-users, but a release can’t hurt, and might give us some publicity and/or get people interested. Who knows.
I could use some testers for those binaries, though, to make sure I get the library dependencies correctly. And that the GNU/Linux binaries work on other systems than just mine.
I’m also open for other platforms. Would it make sense to have xoreos pre-compiled for Free/Net/OpenBSD? Other architectures than just x86/x86_64? Anybody with insights there, and capable of compiling those binaries (or pointers to cross-compilers), please, contact us. :)
As for how to continue the actual xoreos development, I think it would be useful to transfer the script system that’s currently hooked up to Neverwinter Nights onto the other engines. It would need to be rewritten, though. When I first wrote it, I wanted to have engine functions with signatures that mirrored the signatures of what the scripts call. I couldn’t get it to work, though, and settled on a context that contained an array of parameters. For some reason, I still used boost::bind for all the functions, which, at that point, was not necessary. boost::bind compiles really, really slow, and so now the files containing the Neverwinter Nights engines functions take ages to compile. This needs to go.
There, that’s the current short-term roadmap for me: cleanup, release, script system.
]]>Now that I had (nearly) everything graphical together, it was time to weave it all together into something approaching fake Sonic Chronicles gameplay.
Being a Nintendo DS game, Sonic Chronicles run on two screens of 256x192 each, arranged on top of each other. To make this easy on me, I decided to, for now, force xoreos’ window size to a static 256x384, and draw into it as if it was the two Nintendo DS screen. For the future, we’ll have to think about how to handle scaling.
There’s at least two ways to handle this:
The former is the easy way out, but the latter might provide higher quality. There might be a third, a middle way: draw all 2D elements combined with scaling, and render the 3D objects in higher resolution.
To make the game feel a bit more real, I rigged up few static panels showing the various splash screens, and the “Start your adventure” GUI. Note that the button isn’t actually working: it’s really just an image that waits for a mouse click.
After the intro panels, Sonic Chronicles in xoreos then dumps you right into the first area. Using a static panel again, it displays the mini map on the top screen, and the area background on the bottom screen. With the arrow keys, you can move the camera along the X and Y axes, and the area background panel follows the camera to draw the section.
That was easy enough. I then wasted the better part of a day trying to trace and guess at how the game loads the “placeable” objects, the usable 3D objects in the area. The area description within the ARE file lists all placeables, each with an integer with the “name” of 40023 that seems to be a running ID and an integer called 40018 that’s unique for each type. I.e. collectable rings have a 40018 value of 0, the item chest 6, and the pile of wood in the first area has a value of 15. The model names are listed in appearances.gda. So far, so familiar. However, the model for the wood pile is on row 101, and I failed to find a consistent way to connect those two numbers, 15 and 101, either mathematical or with the help of other data files, that would have worked for other placeables as well.
With nowhere else to turn, I looked at the disassembly. And I wept. There is no clean way to connect those numbers, because the placeable instantiation is hardcoded: there’s a big old switch with all possible values (43 of them, 0-42) for the integer 40018, with object instantiation for each of them.
To keep it simple for now, I added a little array mapping that type ID onto a row in the appearances.gda. Not all cells are filled yet, but enough that the first area makes sense.
But, to get the models to display correctly, there was still something missing. You see, xoreos sets up a proper perspective projection matrix, where objects in the distance are smaller and all this jazz. Sonic Chronicles, however, uses an orthogonal projection at an angle of 45°. So, I added a method in our GraphicsManager to let the game code select an orthogonal projection instead.
And after some other minor fixes related to this, like scaling the rate of camera movement to fit the 45° angle (so that 3D objects stay at the same point on the background image when you move), and adding the changed orthogonal viewport to our unproject() function (so that detecting that you’ve moused over an object works correctly), the placeables now display and behave correctly within the areas.
So, what’s missing? Quite a lot, actually:
Model animations. Those can be both geometry- and texture-based. In geometry-based animations, the bones move around and rotate, leading to different vertex positions. With texture-based animations, the textures move, rotate or scale, or even get replaced by different textures.
Most of the placeable types aren’t yet recognized. Nor do the placeables do anything yet. Nor do we create creatures, triggers and squads. Nor do we have a player character that can move around.
There are also no conversations of any kind implemented yet, and there’s no proper GUI and menu support.
We’re also lacking sound and music. There’s partial documentation for them, though, so it should be relatively easy to manage. Videos, which we still miss too, will be more difficult: we need to reverse-engineer the ActImagine VX video codec, since no one has done that yet.
This all is stuff for the TODO pile, though. Nothing I want to work on at the moment. Of course, we would welcome your contribution, so please, don’t hesitate to contact us if you want to tackle any of these features, or anything else for that matter!
If I want to continue on the path of getting all games to show areas, Dragon Age: Origins would be next. We’ll see how well I’ll do there, I guess! :)
]]>After having implemented readers for the common BioWare formats, I turned to the graphics formats. They’re, for the most part, stock Nintendo DS formats, which means I could build upon detective work from the Nintendo modding scene. I have to thank three people in particular: Martin Korth, of NO$GBA fame, whose GBATEK documentation is invaluable, lowlines who figured out much of the gory details of Nintendo’s formats and pleoNeX, whose GPLv3-licensed tool Tinke provided the base on which I implemented my code.
When I looked over the files inside the Sonic Chronicles archives, I noticed a peculiar thing. There’s a lot of files with names ending in “.small”. I suspected a compression scheme, especially after examining the files in a hexeditor. And sure enough, there are several compression algorithms provided by the Nintendo DS firmware. The one used by Sonic Chronicles is an LZSS variant, which can be decompressed using barubary’s MIT-licensed dsdecmp tool (GitHub mirror). I pulled the decompressor into xoreos.
The first graphics format in Sonic Chronicles I inspected was NSBTX. NSBTX files are texture; or rather: archives of several textures used by a single model each. Implementing the reading of these was simple enough, and I added a small program to our tools collection that can “extract” them into TGAs.
Next up, I wanted to see the fonts, NFTR, used in the game. They’re bitmap fonts, with each glyph an image. The image can be 1-bit black and white, or it can be greyscale for anti-aliasing, shadowing or outlining purposes. Additionally, there’s mapping tables to translate a code point in a certain encoding (UTF-16, UTF-32, CP1252 or Shift-JIS) into the index of the glyph it represents.
There was a bit of trial and error involved, as the documentation and existing FLOSS projects to display the fonts weren’t quite correct in certain details (which might not even be their fault; Nintendo likes to subtly change formats between firmware versions). But, before long, I could print arbitrary strings in these fonts in xoreos.
Sonic Chronicles comes with a few NBFS files, which is a dead-simple raw format: 8-bit paletted image data, with the palette (in 16-bit RGB555 values) in an NBFP file of the same name. They’re mostly used for images spanning a whole Nintendo DS screen.
The main image format used in Sonic Chronicles, however, was still missing: NCGR. As I went along implementing a reader, I learned these ugly facts:
Essentially, this image of Sonic is divided into these parts:
This all makes assembling the final image a bit…ugly. But hey, I made it work in the end.
…except for one thing: a few of the NCGR files don’t specify their width and height. By fiddling with the values a bit, I managed to find these values manually, but the resulting image looks off: it’s as if the image is supposed to be rearranged afterwards, different pieces drawn at different places. Each of those file also has an NCER file with the same name. I assume that means information on how to draw these NCGR are within the NCER. A thing for the TODO pile.
I then went for the big one: the 3D model format NSBMD. This involved a lot of fiddling, guessing and trial-and-error, as the documentation of these formats is sparse, and oftentimes wrong.
Conceptually, an NSBMD file consists of these parts:
A bone consists of a name and a transformation that displaces it from its (at that point unknown) parent bone. They are stored as a flat list. The bone commands then specify how the bones connect together. And they give each bone a location in the Nintendo DS’s matrix stack (a list of transformation matrices containing the absolute transformation of each bone at the time of rendering).
The material contains the texture name (which reference textures inside the NSBTX with the same name as the NSBMD), and a few properties.
Each polygon can use a single material, and contains a list of polygon commands. These polygon commands produce the actual geometry. They set color, normal and texture coordinates, and generate vertices. They also manipulate the matrix stack, specifically replacing the working matrix with the matrix from the stack position of certain bones. In essence, this bases the vertices on the position of the bone.
While the Nintendo DS interprets the polygon commands on-the-fly while rendering, and while they can be nearly directly converted to OpenGL-1.2-era glBegin()/glEnd() blocks, this is not really want we want to do. So instead, we, while loading, interpret the polygon commands into an intermediate structure.
The result is a relatively massive loader for these files, and that doesn’t yet include support for animations.
One interesting anecdote: the Nintendo DS doesn’t use floating-point numbers to represent real numbers, but various formats of fixed-point numbers. Those are found extensively in the NSBMD files. But when I implemented the GFF4 format earlier (see part 1 of my Sonic Chronicles progress report), I found, in the GFF4 files used by Sonic Chronicles, a field type not described in the Dragon Age toolset wiki. Turns out, those are Nintendo DS fixed-point numbers!
With those pesky models out of the way, I was ready to show the areas, right? Wrong. There’s yet another graphics format in Sonic Chronicles: CBGT, used for the area background images.
However, CBGT isn’t a Nintendo format. No, it’s one of BioWare’s creation. It does, though, take inspiration from the Nintendo DS formats. It consists of blocks of 64x64 pixels, each compressed using the LZSS algorithm found in SMALL files, and each block divided into 8x8 pixel tiles. PAL files of the same name carry palettes, with each CBGT able to use a different palette within the PAL.
Since I already knew how to puzzle together those cells and tiles from the NCGR format, getting the image itself was not a problem. But I was at a loss where to get the dimensions of the image from, and how to distribute the palettes onto the cells. I figured out an algorithm for the latter, that worked for nearly all images, but the outliers still annoyed me. Then it hit me: for each CBGT/PAL pair, there’s a third file: a 2DA. And that one contains the information which cell uses which palette, neatly organized in a 2D table exactly how the cells are arranged in the final image. This, of course, is enough to calculate the final image dimensions as well.
I also found a fourth file for nearly each CBGT/PAL/2DA tuple: a CDPTH. Arranged in a similar fashion to the CBGT, it contains 16-bit depth information for each area background. This is used to let certain background pieces draw over the 3D models in the game, when they should appear behind something.
Now I was ready to implement actual Sonic Chronicles stuff. I’ll describe that in part 3.
]]>Yes, a Nintendo DS game. I wasn’t so sure myself that game is actually a “proper” target for xoreos. I’m still not 100% sure, but I know now that it at least does use several BioWare file formats, as well as Nintendo DS formats. I also saw that some of those BioWare formats are used in Dragon Age: Origins as well, so Sonic Chronicles actually did provide a natural station on my path.
I’ll divide my report in three parts. In this post, I’ll go a bit into the details of those common BioWare file formats. In the next post, I cover the graphics (that are mostly Nintendo formats). And the third post will show how I tied it all together in xoreos.
So, onwards to the BioWare formats.
GFF is BioWare’s “General File Format”, which is used as the basis for many things in BioWare games. It’s an old format, already found in the Infinity Engine, but not quite as complex yet. (Correction: It seems I misremembered there; GFF is not used in the Infinity Engine. I apologize for this mistake.) Conceptually, it is comparable to XML1: hierarchical data, organized in a tree-like fashion, able to hold basically everything. As such, it’s used to describe areas, characters, items, dialogues, … Unlike XML, however, GFF is a binary format, not directly human-readable.
Since GFF is such an important format, xoreos already implemented a reader (thanks to BioWare releasing specifications for the Neverwinter Nights toolset. And we provide a tool to convert them into XML for easier readability, too. It was only, however, for versions 3.2 (used by Neverwinter Nights, Neverwinter Nights 2, Knights of the Old Republic, Knights of the Old Republic 2 and Jade Empire) and 3.3 (used by The Witcher). But Sonic Chronicles, Dragon Age: Origins and Dragon Age 2 needed a reader for versions 4.0 and 4.1 – and boy did they change the format.
You see, after converting the GFF3 to XML, the whole thing is really quite readable and understandable. Every tag has a full string as a name, making the uses and intentions clear. But from the game’s perspective, this has a huge drawback: it’s slow. Strings are unwieldy, slow to read and compare, and variable length items are generally a pain when you want to quickly jump to a specific field. To curb that, GFF4 removes those pesky strings. Instead, fields use a single 32-bit integer as their “name”, making comparisons easy as pie.
Lucky for me, the new GFF4 format is already documented on the Dragon Age Toolset Wiki. The huge amount of example files provided by the two Dragon Age games and Sonic Chronicles gave me ample opportunities to test out corner cases as well. Easy. The gff2xml tool mentioned above now supports GFF4 as well.
[1] In fact, BioWare generates their GFF4 files out of XML, as can be seen from the Dragon Age: Origins toolset.
Next up, I saw a new TLK format used in Sonic. TLK is a “talktable”, a list of strings indexed by a numerical ID. The idea is that you have all text used in the game in one place, easy to use and easy to translate. Already used in Neverwinter Nights, xoreos has a reader for it already. It’s relatively simple, too.
However, the new format is quite different. In fact, it’s a GFF4! I did say that you can basically stick everything in a GFF, right? That’s what they did for Sonic Chronicles (and the two Dragon Age games). With the new GFF4 reader, adding GFF4’d TLK support was quick and painless.
Just like the GFF4’d TLK, GDA is an old friend in GFF4 suit. This time, it’s 2DA: a 2 dimensional array, a table if you will. If you’re still lost, think Excel spreadsheet, a simple collection of data organized on a grid.
2DAs are used to, for example, specify the models of different objects. The GIT file describing objects in an area would say “Here’s an object, we call it Chair, it has Appearance 179”. The game then looks into appearances.2da, at row 179 and column “ModelName”, grab that filename there and load it as the object’s model.
GDA is, essentially, just the same thing as GFF4. A list of columns giving their name and type, and a list of rows with the data for each column. However… While real 2DA have an actual column name (the “ModelName”, for example), making guessing the meaning easy, GDA don’t actually store a name. They store a hash of the name (specifically, the CRC32 of the UTF-16LE encoded string in all lowercase), a number that’s meaningless in and of itself.
There’s 845 unique hashes in the GDA files found in Sonic. There’s no real way to turn them back into readable strings, but there’s a certain trick I could apply: a dictionary attack. I got myself a huge list of words found in a dictionary, hashed them, and compared the hashes. Then I extracted all strings I could find in the game (from the GFFs, mostly), and did the same. Then I combined the words of these lists. Then I combined matches. Each time, I manually went through the list to kick out the many, many false positives: strings that hashed to a valid number, but that don’t make sense in the context of the game (“necklessnoflyzone”, “rareuniquemummifications”, “properlyunsmoked”).
Phew, that was a lot of tedious work. Still, I managed to find the source strings for 534 of those 845 hashes, 63%. Sure, there’s still 311 missing, but that’ll have to wait for later.
And that’s it for the common BioWare file formats. Tune in next time when I go over the graphical formats.
]]>As some people might know, CD Projekt RED licensed BioWares Aurora engine for the first The Witcher game (and only the first; the later parts use their own REDengine). And in many aspects, it’s very obvious that they spawn off directly from the Neverwinter Nights codebase, not from later BioWare titles. There have been quite some changes, though, to accommodate for The Witcher’s requirements.
Due to their similarity to Neverwinter Nights‘ MDL model files (and with light help from Michael_DarkAngel’s twMax code), I had already a partially working loader for The Witcher’s MDB files. Or so I thought. Turns out, my quick & dirty hack was just about enough to get simple object and character models to show, but it totally failed on area geometry models. Those turned up completely invisible.
Closely looking at what my code does made me realize that the geometry itself loaded, but the textures failed to load. In fact, the textures it tried to load didn’t exist. However, there were similarly named ones in the resources. In fact, for the requested texture “foo”, there were texture resources “foo!d”, “foo!n” and several others.
Well, these are, it seems, lightmaps, and several different ones depending on the time of day (named after their Polish words). There’s:
Since not all of them might exist for a given texture, I settled on just loading the first one available. And yes, that gave me textured area geometry. With just the lightmap applied, it still looked a bit low-res, however. No wonder, there needs to be a base texture as well. Unlike Neverwinter Nights, which just straight up names the textures and has simple TXI files for some texture properties, The Witcher has full-fledged material definitions integrated into the model. And they’re shader-based. For the object models, it was enough to just take the texture names and run with it, but for the area geometry models, I had to extend this a bit. Granted, this is still a hack (we still don’t support shaders), and fails occasionally, only less so than before.
The result was this:
There’s bits missing, I hear you say? Correct. Unfortunately.
More debug printouts on the model loader clued me in: there’s a new kind model node! The twMax author also noticed this; he calls them “TexturePaint”. No support for them in twMax, though. My Google-fu didn’t uncover anything else helpful either.
With no existing tools to help me, I had to do the dirty work myself. I pulled out my trusty friend the disassembler. Luckily, CD Projekt RED kept the tradition of supporting ASCII representation of model files, and so I was able to map out the loader code relatively quickly, for the most part.
First, I filled out my loader to stub and comment more MDB fields I previously just ignored. Then, I started implementing the TexturePaint nodes. Thanks to a brief email conversation with someone who’s also playing around with The Witcher models, I already knew what these nodes probably represented: geometry textured by blending several distinct textures together, to create more realistic terrain. I.e. similar to what I found Neverwinter Nights 2 does for its terrain geometry.
This turned out to be exactly the case, only with in-node weightmaps instead of the color channel approach in Neverwinter Nights 2. Additionally, these nodes too have a lightmap applied. Without shaders, this mixing is awfully slow and memory-consuming to do, therefore I instead just apply the lightmap for now. The result looks like this:
It’s not exactly pretty, but it at least shows something.
After having implemented the TexturePaint nodes, I found a curious issue: certain nodes appeared twice, overlapping each other:
This is the result of the LOD information in the node headers. Some nodes are supposed to be displayed when you’re near, some when you’re far. Simpler textured geometry nodes are displayed instead of the more complex TexturePaint nodes when far enough away to not notice the texture blending anyway. As a workaround, I rigged it to only display the highest LOD for now.
With the area models correctly loading, I went on to actually load the area. Owing to its origin, it’s again very similar to Neverwinter Nights, with one difference: no tiles and tilesets. Instead, the “tileset” value specifies the singular area geometry model to display.
I quickly implemented loading said area geometry model and simple area objects. And was baffled. The area geometry’s position didn’t match up with the objects’ positions. The area was so far in the distance, you couldn’t even see it. I looked and searched for a “tile” position in the area description files…nothing. As an experiments, I bound moving the area model to keyboard keys, and played around with it until it fit. The correct position, for some reason, is {1500.0, 1500.0, 0.0}. Don’t ask me why, but this works for all areas in the game. ¯\_(ツ)_/¯
There was another thing I noticed, though: the orientation of the area objects is wrong. Unlike Neverwinter Nights, which only specifies one angle, the “bearing”, for each object, The Witcher lets you rotate all objects in all three axes. The orientation is described as a quaternion. Now, Neverwinter Nights 2 does the same and because we need the object orientation in Euler angles instead, we convert them. Unfortunately, that code doesn’t seem to work correctly in The Witcher. I assume it’s connected to the fact that The Witcher actually fully rotates the objects (while Neverwinter Nights 2 only, in effect, rotates around two axes), combined with how xoreos changes the axes around for world objects and additionally needs to translate the coordinate system from Direct3D orientation to OpenGL’s.
Try as I might, I couldn’t get the correct orientation. After way too much banging my head against the wall, I caved and put that onto the TODO pile. That’s something I have to revisit another day.
“Only objects? Where are the creatures, the NPCs, the people?”, I hear you ask, imaginary reader. Well, The Witcher doesn’t directly describe creatures in the area files. Instead, there are spawn points, and I think the rest is handled by the game scripts. Not something I can do now. No NPCs for now, I’m afraid.
A few bits and pieces I found out or did during this endeavour:
In conclusion, area loading in The Witcher is now in a similar state to area loading in Neverwinter Nights, Neverwinter Nights 2, Knights of the Old Republic, Knights of the Old Republic II and Jade Empire. It’s not flawless, and there’s still a lot of things missing, but it’s a start. :)
What’s next? There’s three games left without area support: Dragon Age: Origins, Dragon Age 2 and Sonic Chronicles: The Dark Brotherhood. The latter is a Nintendo DS game, and is missing basic support for a lot of DS-specific file formats. In fact, I’m still not 100% sure this game even belongs in xoreos… The two Dragon Age games are somewhat further along: resource loading works and the texture format is known. They are, however, completely missing model support, as well as support for the new GFF version. Since that is at least something to go by, it’s possible I’ll tackle Dragon Age: Origins next. It could take a while until I have something worthwhile to report, though.
]]>So, with no other place to look, I buckled down and opened the Jade Empire binary in a disassembler.
The loader function was quickly found, and in combination with Maian’s findings, the header was a cakewalk. The model nodes, the structures containing the mesh data, however, proved to be more tricky: the engine simply read the whole shebang into memory, to be used later.
Trying to find a shortcut, I remembered that Neverwinter Nights was able to load ASCII representations of its models. I searched for some common keywords like “verts” and “faces”. I had luck: there still is a ASCII model loader in Jade Empire, and not just a remnant of older code; it was updated to load ASCII representations of Jade Empire models. Moreover, for the most part, it actually parsed the model values into the same struct it reads the binary data into. Meaning: I could directly map data from the model file to their ASCII name, getting their meaning handed on a platter.
This basically gave me the entirety of the general node header, and a good part of the TriMesh node (that contain model geometry in form of a triangle mesh) header. What it did not help with was the format of the vertex data and face indices; those were parsed into a different structure.
Chasing down the place where the binary data was put into that structure wasn’t too difficult, and soon I had enough information to render basic, untextured geometry:
While there was a texture field in the TriMesh node, I found it was more often then not empty. Instead, there was a numerical material field, and a matching .mab file in the resource. A material definition in binary format. I also found several .mat files in the resources, ASCII material definitions. Trying my luck again for finding a loader of these in the binary, I searched for the string in the disassembly, and yes: there is a loader for the ASCII material files, parsing them into exactly the layout of the binary .mab files.
With that, and some fiddling with the offset into the vertex structure for the texture coordinates, I managed to render textured geometry:
People familiar with Jade Empire might recognize that model: that’s Silk Fox (SPOILER WARNING on that link!). And for some reason, she usually wears a veil in front of face. So, where’s that veil?
Turns out, while meshes in Knights of the Old Republic’s models are always triangle lists (simple, unconnected triangles), Jade Empire models also contain other mesh types: triangle strips and triangle fans. For the sake of simplicity, I decided to unroll them into triangle lists for now, and that made the veil visible:
I said above that I fiddled a bit to find the texture coordinates inside the vertex structure. Well, in fact, what I did was a big, fat hack: I hard-coded the offset based on the size of the structure. Obviously, that’s wrong. And it did fail for a lot of models, most prominently the models used for the area geometry.
Wasting quite some time trying to find how the vertex structure is interpreted in the disassembly (and even trying to trace it with WineDbg), I eventually found the answer where I should have looked way earlier: in the Knights of the Old Republic model loader. Since the two model formats are similar, I could just apply what the one did to find the offset in the other. With that, the area geometry renders more correctly:
Please note a few things. Yes, not all fields in the material definition are used yet (in fact, only the 4 base textures are used). As such, the rendering looks a bit off, especially where environment mapping should happen. That’s why the roof is semi-transparent, for example.
Also, while I did find the offsets for the textures, I’m not yet sure about the other pieces of data. Specifically, the normals and data found after the texture offsets. For example, take a look at this hexdump of the vertex array for Silk Fox’s head (click for a bigger view):
Each colored section is a new vertex. The red box, i.e. the first twelve bytes of each vertex, is the vertex position, stored as 3 IEEE floats in little endian byte order. The green box, i.e. 8 bytes at offset 0x14 within each vertex, is the first set of texture coordinates, stored as 2 IEEE floats.
Now, the field that in Knights of the Old Republic models denote the offset to the normals, points to 0xC. Unfortunately, there’s only space for 2 floats here, while the normal usually needs 3 floats. I have frankly no idea what’s going in there. In the case of the first vertex, the data could intentionally overlap, since the 0.0f would fit, but that breaks down not much later.
The data after the texture coordinates looks pretty un-floaty, more like several uint16 values. Again, no idea what that could be yet.
I am, of course, always open to suggestions if you should know what this could mean. :)
In either case, this brings Jade Empire model support close to the other model formats. I found that enough for now; I did spent about 50 hours staring at disassembly to get this far, after all.
Like a lot of things, areas in Jade Empire are structurally similar to areas in Knights of the Old Republic. A .lyt file specifies the layout: the positions of several models (called “rooms”) that make up the area geometry. And a .vis file specifies which rooms is visible from where, so that, in-game, rooms that are not visible from the current location can be culled from the render queue.
Unlike Knights of the Old Republic (and also unlike Neverwinter Nights), the Jade Empire areas don’t have .are and .git files to describe static and dynamic elements of the area, like the name, the music playing in the background, and objects and creatures found within. Instead, there’s another array in the .lyt file for simple, state-machine driven objects like doors, and an .art file gives some general information for each room (mostly wind, fog, camera perspective). Everything else seems to be placed or started by the game scripts.
A script system is not something I want to bind to the Jade Empire engine just yet, but loading the layout file and placing the room models, that I can do:
Again, the same caveats as above apply: no proper material support yet. And due to these issues of missing environment mapping and wrong blending, those two screenshots are actually less ugly than others. See for yourselves:
Still, apart from the missing NPCs and objects, Jade Empire areas are now in a similar state to areas in Knights of the Old Republic. That counts for something, no? :)
]]>As a result, areas in Neverwinter Nights 2 are now in a similarly supported state as areas in the two Knights of the Old Republic games.
So, here’s a few interesting things and/or issues with them.
Like the first Neverwinter Nights game, areas in Neverwinter Nights 2 are (or can be, more about that later) made up of tiles. Meaning, a grid of models define the general structure: piece of indoors floor with walls, corners, windows and/or doors. Unlike the first game, though, there’s also metatitles, which are bigger models spanning the space of several regular tiles. This is used, for example, for the piece of the cave where you wake up in the Mask of the Betrayer expansion.
xoreos correctly loads and displays both tiles and metatiles.
Also unlike the first game, areas in the second game can also use terrain instead (or maybe even additionally to) tiles. The terrain is floor geometry unique to the area, and consists of patches of land and water, together with walk mesh definitions. Terrain is used very often in outdoor areas, to give those areas a more unique and less flat feeling.
xoreos correctly loads and displays the land and water patches, in a very rudimentary and stubby way.
Neverwinter Nights 2 adds a new area object type: environmental objects. These seem to be quite similar to regular placeables, only that they are static and can’t ever be moved. Apparently, their geometry is also baked into the area walk meshes.
As an extra weird bit, environmental objects store their position and orientation differently than all the other objects.
xoreos correctly loads and displays environmental objects in areas.
Neverwinter Nights 2 also adds new, non-static trees, courtesy of the SpeedTree middleware. Tree definitions seem to consist of just a few parameters and a random seed, the rest is done by the SpeedTree library.
xoreos does not load or display those trees yet. In fact, that will be a major task in the future. Luckily, trees aren’t essential; them missing is not a showstopper.
xoreos also partially loads and displays the various creatures (animals, NPCs, monsters, …) defined in the area files. There’s one caveat, though: the different model parts (head, hair, gloves, boots) aren’t actually attached to the body yet. They’re loaded as separate models and those are already placed correctly for default character dimensions. Of course, this would fail with scaled models (which we do not yet support).
The underlying problem is this: where the character models in Neverwinter Nights and Knights of the Old Republic contain special nodes telling the engine where to place the different body parts, Neverwinter Nights 2 has no such thing. Instead, each skin geometry vertex has a “bone index” attribute that, I assume, are indices into the Granny 3D animation files.
Which means: to properly attach the body parts, and to animate the models, we first need to figure those files out.
A lot of objects in Neverwinter Nights 2 areas use tint maps. These are special textures with intensity values in their color channels. When an object is loaded from the area definition, those also contain 3 color values, and those color values are combined with the intensity values in the tint map to create a unique tinting of the object. This gives you the ability to create blue-ish caskets with yellow-ish metal rings, purple carpets, green vases and the like, without having to manually create textures for each color combination.
Moreover, tiles also use this tinting.
Currently, xoreos can evaluate and use those tinting, but entirely in software. For each object, an uncompressed RGBA texture is created, and the color values are mixed on loading. Of course, this has certain drawbacks:
First of all, it’s slow. But more damning, it takes up a lot of extra memory, both on the GPU and in the system’s RAM. As a ballpark, a common large-ish area creates 1GB of extra textures; even more when there’s a lot of objects around.
This is, of course, unacceptable. In the future, this tinting should most probably be done with shaders on the GPU.
There is another tinting mechanism: terrain geometry uses a similar approach, only more extended. Instead of just 3 mix channels, there’s 6 (using 2 mixing textures) and instead of just mixing colors, textures are mixing. I.e.: a terrain patch can specify up to 6 texture that will be mixed according to the color values in 2 mix textures. The result is a layered texture that removes all visible texture tiling effects.
Additionally, each patch also has a color value to modify the texture, increasing the flexibility again.
At the moment, xoreos does none of this, for obvious reasons. Instead, the singular color value is applied to each patch, creating a more flat look.
Currently, xoreos does not display most doors and chests entirely correctly: doors are missing their, well, door part (only the door frame and knob are visible) and chests are missing their lid. I have no idea if that’s connected to the animation system, tinting, or what.
All in all, I’m pretty happy with the progress so far. :)
]]>If a story-driven Neverwinter Nights 2 persistent world is your thing, please have a look at their’s, Realms of Trinity, and consider supporting them. Thanks.
The Neverwinter Nights Podcast website might go offline in May 2015, so I took the liberty of mirroring the episode with my interview here. You can find the original text accompanying the interview below.
Please join Mark AKA Madmage999 and Brian AKA Sir Brian, from NWN2 Realms of Trinity, as they interview the lead developer of the Xoreos project, Sven AKA Dr. McCoy.
Neverwinter Nights Podcast Episode 192
xoreos is an ongoing FLOSS project to reimplement BioWare (and derivatives) as used in their 3D games, beginning with Neverwinter Nights. The goal is to have a portable program to play those games even on operating systems those games were never meant to be played on, such as GNU/Linux and Mac OS X.
xoreos is free software and licensed under the terms of the GNU General Public License version 3 or later (GPLv3+).
To use xoreos to run the games, you have own the original game media. We do not endorse piracy in any way, shape or form.
In its current state, xoreos is still far away from its goal. At the moment, we are able to load resource archives and read basic file formats. Video and sound playing works, models are partially supported, and for earlier games, areas are shown and even some very basic gameplay exists.
A lot of work is still left to do. We are actively looking for developers interested in joining our efforts in taking apart BioWare games and reimplementing them. If that sounds something you would find exciting and feel capable of, please contact us. We are always happy to see new faces. :)
The most pressing matter, however, is finding a person with knowledge of OpenGL: The current graphics subsystem is barebones and was quickly hacked together without much deeper understanding. If works for what is shown right now, but is far from being anywhere near adequate. There are a lot of features missing, and the performance is nothing to be proud of. Again, we would be very glad to welcome you in our still small team!
]]>So, let’s start with what has happened: there has been quite some work and changes and fixes under the hood. Not as much as I’d like, and no hugely visible new features yet, but still.
On the code side, we have:
Additionally, I started working on another little tool: Phaethon. Phaethon is a graphical resource explorer (using wxWidgets) for BioWare Aurora engine games, able to look into the archives, extract files, display images and play audio files. It’s not yet finished, and there’s several things left to implement, including displaying GFF files. The goal is to have a more user-friendly, GUI companion to the existing CLI xoreos tools.
There has also been several improvements on the documentation side of things:
As you can see, this is quite a list. And that’s only half of the story. Leading to what (or, more precisely, who) I am thankful for, there’s been contributions by several incredible people:
There, this is what’s happened in the last 7 month since the last blog post; this is the current state of the project. Looking forward, here’s hoping that we all continue to find time to work on xoreos, that maybe more people will join us and that together, we can make this project bloom into something wonderful.
]]>Not being used to being interviewed, or speaking English for that matter, my nervousness is quite audible. But I hope I still got my points across. :)
So, if you’re interested, give it a listen.
]]>This will hopefully be more flexible than the GitHub wiki, and will allow easy contributions by people outside of GitHub. The idea is that this will become a knowledge hub into the internals of BioWare’s Aurora engine. Any interesting things anyone has found out, we’ll be glad to have documented here. So please, if you feel the need to crawl deep into the bowls of the Aurora engine, or have already done so, add your newly-found understanding to our wiki. :)
We will be migrating the GitHub wiki pages to our wiki, and see about adding some new information as well, in the following days.
]]>