As a belated 10th birthday (yes, xoreos is 10 years old now), here’s a new release of xoreos! It’s xoreos 0.0.6, nicknamed “Elanee”, including xoreos, xoreos-tools and Phaethon.

In xoreos proper, the most visible change is in Knights of the Old Republic: seedhartha has put a lot of work into making the tutorial partially playable! The player character can collect their belongings, put on their gear, let Trask open the first door, open the second door and engage in mock combat.

You can see it in action in this video here:

Under the hood, the code for Knights of the Old Republic and Knights of the Old Republic II was fundamentally restructured to build off a common base, so a lot of the underlying concepts in the first game are also available in the second, but are not as visible. You can, however, skip the prologue in Knights of the Old Republic II to watch your player character awaken in the Peragus mining facility and trigger a few early interactions.

Watch it in this video here:

The only other game where something has happened visibly is Jade Empire: you can now run through a preliminary character generation and listen to area background music.

User-invisible changes include:

  • Partial implementations of ActionScript and Scaleform GFx for Dragon Age menus, thanks to Nostritius
  • Partial implementations of XACT WaveBanks and SoundBanks for Jade Empire
  • Partial implementations of FMOD SampleBanks and Events for Dragon Age: Origins, the latter thanks to Nostritius
  • Partial implementations of Wwise SoundBanks for Dragon Age II
  • WebM (Matroska + VPx) support for the Enhanced Edition of Neverwinter Nights, thanks to clone2727
  • Support for big-endian GFF4 files found in console ports
  • Support for resource files found in mobile ports
  • Support for more Neverwinter Nights 2 structures, thanks to rjshae

xoreos and xoreos-tools now require a C++11-capable compiler, in an attempt to modernize a bit. This should hopefully not be a huge problem. Phaethon already required a C++14-capable compiler the last release; this has not changed.

This leads us to the news in xoreos-tools: the xoreos-tools package has gained nine new tools, most of them thanks to Nostritius. These tools are: unobb, untws, rim, keybif, tws, fixnwn2xml, xml2gff, fev2xml and ncsdecomp.

The first two new tools, unobb and untws, are new archive unpackers. unobb extracts “obb” files found in Aspyr’s Android and iOS ports of the BioWare games, which can be either plain ZIP files or, more interesting, a virtual filesystem. untws extracts save files from The Witcher.

The next three tools, rim, keybif and tws are archive packers. rim is the counterpart to unrim, creating RIM archives. keybif is the counterpart to unkeybif, creating KEY/BIF archives (and lzma-compressed KEY/BZF archives found in Aspyr’s mobile ports). However, V1.1 files for The Witcher are not yet supported. And lastly, tws, is the counterpart to the new untws tool, creating save files for The Witcher.

Next up, fixnwn2xml takes the non-standard XML files in Neverwinter Nights 2 and turns them into valid, standards-compliant XML files. This tool is thanks to rjshae, based on asr1’s work, reworking it to fit better into our stream classes.

xml2gff is the counterpart to gff2xml, taking an XML file and turning it back into a GFF. Only GFF3 (GFF V3.2/V3.3) are supported for now, so neither Sonic Chronicles: The Dark Brotherhood nor the Dragon Age games (which use GFF V4.0/V4.1) are supported at the moment.

Another work-in-progress tool is fev2xml, which reads the FMOD event file format FEV and creates a human-readable XML file. Only the FEV1 version is supported and then only a fraction of its features.

Likewise, ncsdecomp is the start of an NWScript bytecode decompiler, built on the foundations of our NWScript bytecode disassembler. It’s highly experimental and we give no guarantees that it works correctly at all.

In addition to these new tools, there are some new minor features and bugfixes:

  • unerf can now extract ERF V2.1
  • erf can now create ERF V2.0 and V2.2
  • xoreostex2tga now supports animated TPCs and swizzled Xbox SBMs
  • gff2xml now supports SAC files and big-endian GFF4s
  • tlk2xml now supports big-endian GFF4s
  • The character encoding matrix for Jade Empire is now correct

As for Phaethon, the 0.0.6 version of Phaethon is mostly a “maintenance release”, keeping the foundation in sync with xoreos. There are no major new features. There are, however, a number of smaller bug fixes. Also, Phaethon can now open ERF V2.1 files and display swizzled Xbox SBM images. Additionally, BIF files can be inspected directly, even when no corresponding KEY file can be found (this does mean, though, that the filenames are missing).

To sum it all up: xoreos inched a bit further along, gained more initial implementations useful later on and xoreos-tools hopefully increased its usefulness for modders and spelunkers. Still a lot more to do, a long way to go. But we are, after all, in for the long ride. :)

Sources and binaries for Windows, GNU/Linux and macOS are attached to the GitHub release, here for xoreos, here for xoreos-tools and here for Phaethon.

Additionally, the repository and the source tarballs contain PKGBUILD files in dists/arch/, debian build directories in dists/debian/ and spec files in dists/fedora, which can be used to build Arch Linux, Debian/Ubuntu and Fedora packages, respectively. Alternatively, the PKGBUILD files are also in Arch Linux’s AUR (here for xoreos, here for xoreos-tools, here for Phaethon), and we have a Gentoo overlay.

And as always, we’re looking for more developers to join us in our efforts to reimplement those 3D BioWare RPGs. If you would like to help, please feel free to contact us. :)

Another year nears its end. After unfortunately skipping last year’s Not-Thanksgiving post, we are, back to our regular yearly retrospection. For people new to xoreos: this series is a kind of introspective look into what happened with the xoreos project over the past year. I’m not in the USA, so I don’t observe Thanksgiving, but nevertheless, I do want to give thanks to all the great people working with me on xoreos, and supporting us in other ways.

KotOR main menu with Darth MalakKotOR main menu with Darth Malak

I already wrote a combination introspective and release post back in July, which gave some updates since the previous Not-Thanksgiving post in 2016, so let’s see what happened since then:

WIP Dragon Age: Origins main menuWIP Dragon Age: Origins main menu

Quite a lot, right? :)

Of course, there’s still things in progress, among them:

  • mirv is continuing work on the new renderer.
  • Supermanu is working on pathfinding.
  • Nostritius is furthering our ScaleForm GFx reimplementation.

xoreos wouldn’t be what it is without the help of a lot of great people, for whom I am thankful.

  • rjshae, for taking up work on NWN2 in xoreos.
  • asr1, for starting the NWN2 XML fixer, that’s now in xoreos-tools.
  • seedhartha, for working on KotOR and getting the renderer rewrite mergeable.
  • mirv, for doing the renderer rewrite in the first place.
  • Nostritius, for doing a heck of a lot of different things in xoreos.
  • clone2727, for keeping me grounded.
  • Luigi Auiremma, for all his research and tools.
  • Supermanu, for his pathfinding work.
  • Farmboy0, for his help and advice.
  • The people at GamingOnLinux, a great community for, well, Gaming on Linux.
  • Linux porters like flibitijibibo, icculus and Cheeseness, for providing this much-needed service and also being all-around good people.
  • A myriads of other people I probably forgot, because I’m bad at remembering.

Hopefully, the future will bring even more movement into this little project. If you would like to help with that, then please, feel free to contact us! :)

Fashionably (?) late, but still finally there, a new release of xoreos arrives! xoreos 0.0.5, nicknamed “Dawn Star”, coming with xoreos, xoreos-tools and for the first time, Phaethon, an in-progress graphical resource explorer.

A lot has happened in the last two years. A busy real life unfortunately made me miss the yearly “Not-Thanksgiving” progress report last year…so this here will be a kind of combination release post and progress report. Buckle up, this will be a long one. If you’re only interested in the short release notes, move on to the GitHub release pages linked above and again at the bottom of the post. Cool? Cool.

So, continuing from the last progress report in 2016, what exactly has happened?

Back then, I said I was working on unit tests for xoreos, planning to make them public in December. I did that, and then merged them in February 2017. Since then, several more have been added and bugs were found (and prevented) with them. I consider them a huge success and a boon to the codebase. However, the coverage is nowhere near 100%. Pull requests adding new unit tests to existing code are always welcome ;).

Soon after, I took another stab at one variant of the texture format in Jade Empire, used for lightmaps. I had previously tried to figure it out, but failed. But with the added knowledge of the textures in the Xbox version of Knights of the Old Republic, which are swizzled [*], I realized that the lightmaps in Jade Empire are also swizzled. They’re even swizzled in the same way, so I could just reuse the deswizzling code, and voilà, now that texture variant is correctly read.

Swizzled textureSwizzled texture Deswizzled textureDeswizzled texture

([*] Swizzling changes how the pixels of the image are stored. Usually, pixels are stored linearly, from left to right, from top to bottom. This is not necessarily ideal for GPUs, though, because GPUs like it when things that are displayed near each other are also stored near each other (spacial locality). And while the pixels left and right of a pixel are near each other, the pixels above and below are not. Therefore, swizzled textures store pixels in a sort of zig-zag order. The Z-order curve is a popular scheme for that.)

Next up, Farmboy0 and Nostritius both worked on the GUI systems in Jade Empire and the two Knights of the Old Republic games. Jade Empire now has a partially working main menu (with dressupgeekout submitting pull requests to make the “Exit” button work and enabling background music), and the Knights of the Old Republic games gained character creators and partial in-game menus.

Jade Empire main menuJade Empire main menu KotOR character generatorKotOR character generator KotOR2 character generatorKotOR2 character generator

Speaking of Knights of the Old Republic, these two games are progressing quite a bit, thanks to seedhartha. They now have a fully working animation system, PC movement (with walkmesh evaluation based on Supermanu’s work) and even initial dialogues. In fact, the first game already shows a semblance of the first few minutes of tutorial on the Endar Spire, with your bunk mate telling you grab your gear from a chest and equip it.

For the Dragon Age games, Nostritius implemented an ActionScript interpreter. Together with a renderer for Adobe Flash vector graphics (still work in progress), this will be a reimplementation of Scaleform GFx. Scaleform GFx is used for the user interface, like menus and quickbars, in Dragon Age: Origins and Dragon Age II. Note that the vector renderer is not yet in the public xoreos codebase and that the screenshot below is more of a proof of concept.

Dragon Age: Origins main menu (WIP)Dragon Age: Origins main menu (WIP)

On the xoreos-tools side of things, I added two new tools, ssf2xml and xml2ssf. ssf2xml takes a sound set file (SSF), as used by the two Neverwinter Nights and the two Knights of the Old Republic games and converts it into a human-readable XML file. xml2ssf is the counterpart, taking an XML file and converting it back into a game-readable SSF file.

Nostritius then added a third new tools, erf, an ERF packer. It is the counterpart to the already existing unerf tool. Where unerf extracts files from ERF archives, erf takes files and creates a new ERF archive. Unlike the unerf tool, erf currently only supports version V1.0 of the ERF format, as used by Neverwinter Nights, the two Knights of the Old Republic games. Jade Empire and The Witcher.

Also on xoreos-tools, cosmo-ray unified the parameter handling. This massively cuts down on code duplication in the tools and removes a potential source of mistakes.

Moving on to the third package, Phaethon. Phaethon is a graphical resource explorer, able to look into the archives found in the BioWare Aurora games and display several different types of resources. Previous xoreos releases didn’t mention it, but it has existed for several years already now. Of course, it’s still far from finished and could use more contributors.

PhaethonPhaethon

In the past, Phaethon used the wxWidgets libraries for the GUI. But I was never really happy with it; I never managed to get widget placement to work consistently across platforms (or even just across different window managers on Linux). I always hoped to rewrite it in Qt some day. Therefore, I was really quite happy that michaelpm54 did that work for me, rewriting Phaethon to use Qt5.

I’m now far more comfortable providing Phaethon release packages along with xoreos and xoreos-tools. However, due to us using Verdigris to compile Qt5 applications without moc, Phaethon now requires a C++14-capable compiler. My local cross-compilers, the ones I use to create the binary release packages for Windows and macOS (and the chroot I use to build the Linux packages), didn’t have that. Consequently, I had to update my cross-compilers and chroot.

I did that, and now I can create working binary packages for Phaethon on these operating systems. And this also opens up a path to let C++11 features into the xoreos codebases in the future, something which I had been thinking about for a while now (currently, apart from Verdigris in Phaethon, all of the xoreos codebases are C++03-compliant).

On the other hand, also on the packages front, I dropped the openSUSE Build Service (OBS), the service that builds binary packages for a number of Linux distributions. I simply do not have enough time to keep the build specifications there up-to-date. Especially since every time a new distribution release is added, it requires me to invest several days worth of work to get them to build. Anybody willing to take over that task and officially administrate the xoreos OBS specs, please contact us.

Last but not least, there were also, of course, a lot of code quality improvements. Many of which were triggered by the introduction of new compiler warnings flags, new compiler instrumentations and static analyzers. I’m really grateful for all the options I have for checking the xoreos codebase, grateful for all the amount of work done by the gcc and clang/llvm people, and all the services offering free access to FLOSS projects.

And that’s about it for the past two years. Many improvements, but also still many things left to do. There’s several things cooking, but I’d rather not jinx them ;). In either case, I’ll try getting out the next release relatively sooner.

Meeting Bastila in KotORMeeting Bastila in KotOR Meeting the Handmaidens in KotOR2Meeting the Handmaidens in KotOR2

Sources and binaries for Windows, GNU/Linux and macOS are attached to the GitHub release, here for xoreos, here for xoreos-tools and here for Phaethon.

Additionally, the repository and the source tarballs contain PKGBUILD files in dists/arch/, debian build directories in dists/debian/ and spec files in dists/fedora, which can be used to build Arch Linux, Debian/Ubuntu and Fedora packages, respectively. Alternatively, the PKGBUILD files are also in Arch Linux’s AUR (here for xoreos, here for xoreos-tools, here for Phaethon, and we have a Gentoo overlay.

And as always, we’re looking for more developers to join us in our efforts to reimplement those 3D BioWare RPGs. If you would like to help, please feel free to contact us. :)

And again a year is nearing its end. Like last year and the year before, I’d like to turn my gaze inwards.

A lot of things happened with xoreos this past year, albeit most of them hidden and “under the hood”:

  • I wrote about disassembling NWScript bytecode. The tasks I mentioned there are still open, too. If anybody wants to take them up, I’d be happy to explain them in more detail :).
  • We released xoreos 0.0.4, nicknamed “Chodo”. That was the only release of xoreos in 2016. xoreos 0.0.4 included some minor fixes and features for Neverwinter Nights, and the xoreos-tools package included the new NWScript disassembler.
  • In April, I reached a streak of a full year of daily xoreos commits. Due to some real life things, I had to take a break there, though. I’m now again at three months of daily commits, but there is a three-month “hole” between April and August.

GitHub contribution graph in AprilGitHub contribution graph in April

GitHub contribution graph in NovemberGitHub contribution graph in November

  • Farmboy0 fleshed out the Jade Empire engine a bit, mostly in the scripts department.
  • Supermanu implemented a huge chunk of the character generator for Neverwinter Nights.
  • Farmboy0 fixed a glitch in the Neverwinter Nights animation system that has plagued xoreos for quite some time: the animation scaling in various creature models was off. This lead to, for example, the head and arms of elves detaching from the body during the yawn animation.
  • I then implemented a few more animation script functions, too, which is especially noticeable in the intro animation for Hordes of the Underdark. I also fixed a mistake in the keyframe interpolation. This takes care of another glitch in Neverwinter Nights: model nodes rotating the wrong way around.
  • smbas added support for Lua scripts in The Witcher. A lot of the initialization code that sets up the classes and functions The Witcher expects to find is still missing, so nothing obvious is visible as of yet.
  • Farmboy0 moved the window handling from the GraphicsManager into a new WindowManager class, making the code more readable.
  • I fundamentally restructured our build system, or at least the autotools part of it (xoreos can be built using either autotools or CMake). Previously, we used a recursive autotools setup, where make recurses into each subdirectory. This is, unfortunately, pretty slow, among other drawbacks. I changed it to be non-recursive now, with the top-level Makefile instead being created using (recursive) includes.
  • I then introduced various smart pointer templates into the codebase, making it easier to read and easier to keep track of memory allocations.
  • berenm added AppVeyor integration. Like Travis CI (which we already use as well), AppVeyor is a continuous integration service. This means that every single commit to the public xoreos repository will now be built on Microsoft Windows, using Microsoft Visual Studio 2015, in addition to gcc and clang on GNU/Linux (via Travis CI). This ensures that any compilation breakage on these systems is immediately visible and can be fixed at once.
  • GitHub added a new feature, “Projects”, that provide Kanban-like boards of tasks. I took the time to fill the xoreos Projects page with boards for tasks from our TODO list.
  • There were of course also various clean-ups, minor fixes and expanded code documentation.

Animation with glitchAnimation with glitch Animation without glitchAnimation without glitch Animations in the HotU introAnimations in the HotU intro

Additionally, there are several tasks currently being worked on, among them:

  • Supermanu is looking into pathfinding.
  • mirv is still working on rewriting the OpenGL renderer.
  • I am currently writing unit tests for the xoreos codebase, using Google Test. I already found multiple issues, bugs, and corner cases while adding them.

From my side of things, my current plan is to make my unit tests branch public some time in December. I’ll write a small announcement here about it then. A new release of xoreos, 0.0.5, should follow early next year.

As always, this all wouldn’t have been possible without a lot of people. For them I am thankful.

  • Farmboy0, for various fixes, implementations and file format spelunking.
  • Supermanu, for his character generator work and pathfinding research.
  • mirv, for continuing to work on the OpenGL rewrite.
  • smbas, for his work on Lua and The Witcher.
  • berenm, for the AppVeyor integration and CMake knowledge.
  • TC01, for writing a Fedora specfile for the xoreos projects.
  • CromFr, for taking a stab at the walkmesh structure in NWN2’s TRN files.
  • clone2727, for invaluable ideas and corrections.
  • The folks at GamingOnLinux, who continue to be a great resource for all things related to Games on Linux.

I am also thankful for all the people who take the time to explain things to others, people who write interesting, useful or needed articles, and people who provide mentoring and help. Relatedly: a week ago, Stephanie Hurlburt published an article with engineers who are willing to mentor or answer programming/engineering questions. I for one think that’s a really great idea. Please take a look at that article.

And now, let’s see what the next year has in store for us. If you, however, found all this terribly interesting and would like to help with our little project, then please, feel free to contact us! :)

A new year, a new release: we are proud to announce the release of version 0.0.4, nicknamed “Chodo”, of xoreos and xoreos-tools.

In this release, Neverwinter Nights now shows speech bubbles for conversation one-liners, as used for cutscenes, bark strings and short NPC dialogues. Additionally, the premium modules BioWare sold for Neverwinter Nights, including the three that come with the Diamond Edition, can now be properly loaded and started.

Speech bubblesSpeech bubbles

An oversight in the handling of the texture fonts used in Neverwinter Nights and the two Knights of the Old Republic games has been fixed. This oversight broke rendering of certain characters, most prominently of those used in eastern European languages and the “smart” single quotation mark that’s used instead of an apostrophe in some strings found in the French versions.

For xoreos-tools, there’s two new tools: fixpremiumgff and ncsdis.

The first tool, fixpremiumgff, can restore the deliberately broken GFF files found in the BioWare premium modules for Neverwinter Nights. The resulting GFF files can then be edited as normal.

The second tool, ncsdis, is a disassembler for the stack-based bytecode of BioWare’s NWScript scripting language. It supports the scripts of all games targeted by xoreos and can disassemble them into a full assembly listing. It can also produce a control flow graph in the DOT description language, which can then be plotted into an image by using the dot tools from the GraphViz suite.

Moreover, this release includes a lot of user-invisible code documentation and quality fixes, in both xoreos and xoreos-tools.

Binaries for Windows, GNU/Linux and Mac OS X are attached to the GitHub release, here for xoreos and here for xoreos-tools. Additionally, packages for various GNU/Linux distributions can be found on the OpenSuSE Build Service (here for xoreos, here for xoreos-tools) and in Arch Linux’s AUR (here for xoreos, here for xoreos-tools).

Alternatively, the repository and the source tarballs contain PKGBUILD files in dists/arch/ and a debian build directory in dists/debian/, which can be used to build Arch Linux and Debian/Ubuntu packages, respectively.

And as always, we’re looking for more developers to join us in our efforts to reimplement those 3D BioWare RPGs. If you would like to help, please feel free to contact us. :)

As I already said in last year’s retrospective, I want to write a bit about NWScript and its bytecode.

First of all, what is NWScript? NWScript is the scripting language and system BioWare introduced with Neverwinter Nights and used, with improvements and changes, throughout the Aurora-based games. Specifically, you can find NWScript driving the high-level game logic of Neverwinter Nights, Neverwinter Nights 2, Knights of the Old Republic, Knights of the Old Republic II, Jade Empire, The Witcher (in combination with Lua), Dragon Age: Origins and Dragon Age II. This is nearly every single game xoreos targets. The only exception is the Nintendo DS game, Sonic Chronicles: The Dark Brotherhood, which doesn’t seem to use any scripts at all.

NWScript is written in a C-like language and saved with the .nss extension. A compiler then translates it into a stack-based bytecode with the .ncs extension, which is what the game executes. That is similar to how ActionScript in Flash videos works, and how Java, Lua and other scripting languages can operate.

Like C, NWScript is a strongly typed language: each variable has one definite type. Among the available types are “int” (32-bit signed integer), “float” (32-bit IEEE floating point), “string” (a textual ASCII string) and “object” (an object in the game world, like an NPC or a chest). Moreover, there are several engine types, like “event” and “effect”, though which of these are available depends on the game. There are also structs, but in the compiled bytecode, they vanish and are replaced by a collection of loose variables. Likewise, the “vector” type is treated as three single float variables. A special type is the “action” type, a script state (or functor) that’s stored separately.

Additionally, Dragon Age: Origins adds a “resource” type (which, in the bytecode, can be used interchangeably with the “string” type) and dynamic arrays. Dragon Age II in turn adds the concept of a reference to a variable to help performance in several use-cases. For these new features, these two games each add two new bytecode opcodes, something not done for any of the other post-Neverwinter Nights games.

To get and modify the state of the game world, like searching for objects and moving them, and for complex mathematical operations like trigonometry functions, the NWScript bytecode can call so-called engine functions. These are functions that are provided by the game itself; about 850 per game, with some overlap. They’re declared in the nwscript.nss file (nwscriptdefn.nss for The Witcher and script.ldf for the Dragon Age games) of each game.

The original Neverwinter Nights toolset came with a compiler, but a part of the modding community, the OpenKnights Consortium, created their own, free software compiler, nwnnsscomp. Unfortunately, it has a few disadvantages. For example, it always needs the nwscript.nss file and it also only handles Neverwinter Nights. And while there has been several variations that have been extended to handle newer games, many of these are only available as Windows binaries. As far as I’m aware, there never has been a variation that handles Dragon Age: Origins or Dragon Age II. Also, since the code hasn’t been touched for 10 years, it’s difficult to compile now, and it doesn’t work when compiled for 64-bit. For what it’s worth, I mirrored the old OpenKnights NWNTools, with a few changes, to GitHub here.

This nwnnsscomp also has a disassembly mode, which can convert the compiled bytecode into somewhat human-readable assembly. This is pretty useful! I wanted my own disassembler in xoreos-tools.

The steps to disassemble NWScript bytecode are the following:

1) Read instructions

Read the .ncs file, instruction for instruction. An instruction consists of an opcode (like ADD for addition), the argument types (which are taken from or pushed onto the stack) and any direct arguments. For example, an addition that operates on two integers would be known as ADDII. The instructions are stored in a list, one after the other.

2) Link instructions

Each instruction may have a follower, the instruction that follows naturally. For most instructions, this is the instruction next in the list. But certain branching instruction, jumps and subroutine calls, also have jump destinations that may be taken.

3) Create blocks

Group the instructions into blocks. A block is a sequence of instructions that follow each other, with two constraints: a jump into a block can only land at the beginning of a block and a jump out of a block can only happen at the end of the block.

4) Create subroutines

Group the blocks into subroutines. A subroutine is a sequence of blocks that gets jumped to by a special opcode, JSR, and returns to the place from where it has been called with RETN. (In many programming languages, for example C, these are also called functions, but we’re calling them subroutines so that they’re not being confused with engine functions. Subroutine is also often the usual term in assembly dialects.)

5) Link subroutines

Record where a subroutine calls another and link the caller and callee, so that a call graph could be created easily. Likewise, the instructions that start and end the subroutine are also separately recorded.

6) Identify “special” subroutines

There are three special subroutines that we can identify:

  • _start(), the very first subroutine that starts execution of the script. It’s the subroutine that contains the very first instruction in the .ncs file.
  • _global(), which, if it exists, sets up the global variables. This is the subroutine that contains an instruction with a SAVEBP opcode.
  • main(), which is the main or StartingConditional function visible in the script source. If a _global() subroutine exists, this is the last subroutine called by _global(). Otherwise, it’s the last (and only) subroutine called by _start().

7) Analyze the stack

This step goes through the whole script and evaluates each instruction for how it manipulates the stack. Since stack elements are explicitly typed, and instructions that create new stack elements know which type they create (either explicitly, or implicitly by copying an already typed element), both the size of the stack and the type of all elements can be figured out. At the end, each instruction will know how the stack looks before its execution. And for each subroutine, we then know its signature: how many parameters it takes, what their types are and what the subroutine returns.

However, this step only works if we know which game this script is from, because we need to know the signatures of the engine functions. And, unfortunately, this step fails when the script subroutines call each other recursively. The stack of a recursing script can’t be analyzed like this.

8) Analyze the control flow

So far, the script disassembly consists of blocks that jump to each other, with no further meaning attached. To extract this deeper meaning, we analyze the control flow for higher-level control structures: do-while loops, while loops, if and if-else blocks, together with break and continue statements and early subroutine returns. Each block gets a list of designators that show if and how it contributes to such a control structure.

9) Create output

Finally, we create an output. This can be one of three:

  • A full listing, including offset, raw bytes and disassembly. This is similar to the output created by the disassembly mode of the OpenKnights nwnnsscomp.
  • Only the disassembly. This output might be used to reassemble the script some time later, should someone want to write an assembler.
  • A DOT file. A DOT file is a textual description of a graph, which can be plotted into a graph with the Graphviz tool. The result is a clear representation of the control flow in graph form.

As an example, here’s a script from Neverwinter Nights 2: this is the original source, and this is the full listing output, the assembly-only output and the control flow graph.

During this work, I have found a few interesting little bugs in the original BioWare NWScript compiler. For example, this little script, hf_c_skill_value.nss in Neverwinter Nights 2 and its disassembly:

1
2
3
4
5
6
int StartingConditional(int nSkill, int nValue)
{
  object oPC = GetPCSpeaker();
  int nSkill = GetSkillRank(nSkill, oPC);
  return (nSkill >= nValue);
}
1
2
3
4
5
6
7
8
9
  00000017 02 06                      RSADDO
  00000019 05 00 00EE 00              ACTION GetPCSpeaker 0
  0000001E 01 01 FFFFFFF8 0004        CPDOWNSP -8 4
  00000026 1B 00 FFFFFFFC             MOVSP -4
  0000002C 02 03                      RSADDI
  0000002E 04 03 00000000             CONSTI 0
  00000034 03 01 FFFFFFF4 0004        CPTOPSP -12 4
  0000003C 03 01 FFFFFFF4 0004        CPTOPSP -12 4
  00000044 05 00 013B 03              ACTION GetSkillRank 3

Specifically, the int nSkill = GetSkillRank(nSkill, oPC); line is compiled wrong. First, an instruction with the opcode RSADDI is generated, which creates a new integer variable on the stack, for nSkill. Then, the arguments for GetSkillRank are pushed onto the stack, and GetSkillRank is called using the ACTION instruction.

Unfortunately, as soon as the compiler creates the stack space for the local variable nSkill, it associates nSkill with this local variable. So when it’s time to push the parameter nSkill for GetSkillRank on the stack, the parameter to the outer StartingConditional subroutine has already been overruled, and the CPTOPSP points to the new local variable.

This renders the nSkill parameter unused and useless, and GetSkillRank is potentially called with an uninitialized value.

For another example, have a look at this script from Neverwinter Nights:

1
2
3
4
5
6
7
8
9
10
int StartingConditional()
{
  int iResult;
  object oPC = GetPCSpeaker();

  iResult = GetClassByPosition(1,oPC) == CLASS_TYPE_DRUID ||
            GetClassByPosition(2,oPC) == CLASS_TYPE_DRUID ||
            GetClassByPosition(3,oPC) == CLASS_TYPE_DRUID;
  return iResult;
}

It’s meant to check whether the player character is a druid. Since you can multi-class in Neverwinter Nights, it checks whether the character is a druid for the first class, for the second class and then the third class. If any of these return true, iResult will be set to true. To achieve this, a boolean disjunction (“or” operation) is used. As is customary in C-like languages, the boolean disjunction in NWScript is supposed to support short-circuiting: if the first part is already true, the second (and third) checks aren’t even called.

Let’s see how the disassembly graph looks like:

BioWare DisassemblyBioWare Disassembly

The first EQII is the first comparison, and then the block in loc_00000057 is supposed to do the short-circuiting. It duplicates the top-most stack element with a CPTOPSP -4 4 before bypassing the second comparison and jumping to the LOGORII that does the boolean disjunction. Unfortunately, instead of directly jumping to loc_00000080 with a JMP, a JZ was generated instead. And since the top-most stack element was already duplicated and checked with the previous JZ, we know that the true-edge is never taken. It is a dead edge.

This has an interesting consequence. The short-circuiting for the boolean disjunction is broken: all parts are always evaluated before the results are or’ed together. In practice, this doesn’t matter much. It makes the code a bit slower, and any side effects will always happen.

Additionally, if the true edge were ever taken, the stack would be in a broken state. Unlike JMP, JZ consumes a stack element, and so the LOGORII would be missing one of its arguments. Because this is not possible, it doesn’t matter for execution, but my stack analysis dies there. To combat this problem, I added an extra disassembly step after the block generation, the detection of these dead edges. To keep it simple, I only do some simple pattern matching, which is enough for most scripts. There are a few cases where it fails, though.

This bug is present in the original scripts coming with Neverwinter Nights, Knights of the Old Republic and Knights of the Old Republic II, but has already been fixed by the release of Neverwinter Nights 2.

This bug is also not present in the OpenKnights compiler:

OpenKnights DisassemblyOpenKnights Disassembly

As you can see, the branch instruction in loc_00000057 is a JMP, as it should be.

So, to recap, xoreos-tools now has a tool that can disassemble NWScript bytecode, similar to the disassemble mode of the OpenKnights nwnnsscomp, with these added features:

  • Easier to compile.
  • Works compiled as 64-bit.
  • Out-of-the-box support for the NWScript found in all Aurora-based games.
  • No need for a separate nwscript.nss.
  • Support for new array and reference opcodes.
  • Deeper analysis of the stack, to figure out the subroutine signatures.
  • Deeper analysis of the control flow, to detect higher-level control structures.
  • Can create control flow graphs in DOT format.

If you’re interested, the source is available here. Binaries will come with the next release of xoreos and xoreos-tools.

There is, however, still a lot left to do there:

  • Create a decompiler: use the detected control structures as a base to generate C-like NWScript source code.
  • Detect chained instructions: something like “int a = b * c” compiles to a lot of instructions that create temporary stack variables.
  • Detect structs and vectors.

Unlike nwnnsscomp, xoreos-tools is still missing a compiler as well. This is something that would be very nice to have indeed. An assembler, which can take the disassembly output and create a working .ncs file out of it would probably be a useful first step in that direction.

If you would like to help and take up any of these tasks, or any other task from our TODO list, please contact us! :)

The end of the year is approaching fast, and just like last year, I want to use this time for some retrospection.

First of all, what happened in the last year?

  • berenm added support for building xoreos with CMake, by the way of parsing the automake files used for the autotools build system. This way, xoreos can now be built with either CMake or autotools. I was skeptical at first, especially since I harbour no love for CMake, but it is working reasonably well and I am quite happy with it. In hindsight, I was wrong to reject this pull request for so long.
  • I focused on supporting all the different model formats used in the Aurora games, and then I made all the games display their in-game areas with objects.
  • xoreos adopted the Contributor Covenant as its Code of Conduct, in the hopes that it helps foster a friendly and welcoming community.
  • The big one: our first official release, xoreos 0.0.2, nicknamed “Aribeth”.
  • I overhauled the script system, making it more generic. This way, I was able to apply it to all targeted games, except Sonic Chronicles: The Dark Brotherhood (which doesn’t seem to use any scripts at all). This included figuring out and implementing four new script bytecode opcodes: two for array access in Dragon Age: Origins, and two for reference creation in Dragon Age II.
  • I implemented reflective environment mapping for Neverwinter Nights and the two Knights of the Old Republic games.
  • I added a new tool to the xoreos-tools package: xml2tlk, which can recreate TLK talk table files out of XML files created by tlk2xml.
  • With these changes, I decided to push out xoreos 0.0.3, nicknamed “Bastila”.

This is all old news, more or less already discussed in previous blog posts. However, since then, I added yet another new tool to the xoreos-tools package: ncsdis. It’s a disassembler for NCS files, the stack-based compiled bytecode of the C-like NWScript, BioWare’s scripting language used throughout their Aurora-based games.

It basically replaces the disassembler within the old OpenKnightsN WScript compiler, with various added benefits. I’ll write a bit more about this tool in the near future, so for now I’ll just leave you with an example assembly listing it can produce, as well as a control flow graph it can create (with the help of Graphviz). As you can see, it already groups the instruction by blocks and subroutines. It performs a static analysis of the stack (to figure out subroutine parameters and return types) and it also analyzes the control flow to detect assorted control structures (loops, if/else). I plan to grow it into a full-fledged NWScript decompiler.

Additionally, I also added support for BioWare’s Neverwinter Nights premium modules, like Kingmaker, to xoreos.

On the documentation side of things,

  • I added comments and documentation to various files in the xoreos sources, hopefully making them more understandable and useful for potential new contributors and otherwise interested people. Considering how awful my memory is at, this is also a kind of future-proofing.
  • Farmboy0 added “research” subpages for various games on our wiki, filling them with information about their workings.
  • I extended our TODO list considerably.
  • I added an example configuration file, and extended the documentation on the wiki on how to compile and run xoreos.
  • I wrote man pages for each tool in xoreos and for xoreos itself. I also added the former to the wiki.

Phew! This is again a bigger list than I had anticipated. This wouldn’t have been possible without these people, for whom I am thankful:

  • I am thankful to berenm for providing the CMake bindings, despite my grumbling about it.
  • I am thankful to Supermanu, for continuing on chipping away on the Neverwinter Nights character generator.
  • I am thankful to Farmboy0, for working on xoreos’ Jade Empire engine and researching game internals.
  • I am thankful to mirv, for continuing with the huge task of rewriting my naive OpenGL code.
  • I am thankful to Coraline Ada Ehmke for creating the Contributor Covenant.
  • I am thankful to all the people in the different BioWare modding communities, for having figured out many different things already. Skywing for example, who had emailed me a few years ago about certain NWScript issues, issues I recently stumbled over again.
  • I am thankful to fuzzie, for giving me pointers on the NCS disassembler/decompiler.
  • I am thankful to the GamingOnLinux people, who do a lot of work reporting on all sorts of Linux-related gaming news, and who so graciously mirror my xoreos blog posts.
  • I am thankful to kevL, for notifying me of issues with xoreos’ build system on configurations I hadn’t thought about.
  • I am thankful to clone2727, for putting up with rants and ravings.
  • I am thankful to all the people who told me when I was wrong, for example when I wrongheadedly silenced clang static analyzer warnings, without understanding what I was doing.
  • I am thankful to everybody else who gave me hints and tips, taught me tricks and procedure, showed me new things, old things, forgotten things, broken things.
  • I am thankful to all the people who are not angry with me for forgetting them, because they are aware that this is not meant as a personal slight ;).

Now that I have these mushy feelings out of my system, here’s hoping for another great year! :)

And like always, if you want to join our effort, please don’t hesitate to contact us!

To keep things moving following the previous 0.0.2 release, we’re proud to announce the release of version 0.0.3, nicknamed “Bastila”, of xoreos and xoreos-tools.

This release features a working script system for all targeted games, with game scripts being fired for the start of a campaign or module, when entering and leaving areas, and when clicking on in-game object. The singular exception is the Nintendo DS game Sonic Chronicles: The Dark Brotherhood, which doesn’t seem to feature any scripts at all.

The vast majority of engine functions, the functions that are called by the scripts and that do the actual work of tracking and changing the game state, are still missing, though. Per game there are about 850 functions (with some overlap) that need to be implemented. We currently have about 90, per game, of these written and working within xoreos. Moreover, many of the functions still missing depend on features not yet implemented.

Apart from the script system changes, 0.0.3 also comes with support for reflective environment mapping in Neverwinter Nights and the two Knights of the Old Republic games. The “metallic” armor and area parts that were rendered transparent in xoreos are now properly reflective. This can be seen, for example, in the Sith troopers in Knights of the Old Republic, in various plate armor worn by NPC in Neverwinter Nights, as well as the metallic floors on the planet of Taris and the icy wastes of Cania. For Neverwinter Nights, xoreos now also correctly smoothes the vertex normals of (binary) models, so that the metallic effect is not broken by sharp polygon edges.

Semi-transparent maskSemi-transparent mask Plus reflectivityPlus reflectivity Correctly rendered Sith trooperCorrectly rendered Sith trooper

Without environment mapWithout environment map Without normal smoothingWithout normal smoothing Correctly rendered plate armorCorrectly rendered plate armor

On the xoreos-tools side of things, there’s now a new xml2tlk tool that can convert XML files created by the tlk2xml tool back into a talk table TLK file. Please note that, at the moment, only non-GFF’d TLK files can be written, as used by the two Neverwinter Nights games, the two Knights of the Old Republic games, Jade Empire and The Witcher. TLK files as used by Sonic Chronicles: The Dark Brotherhood and the two Dragon Age games can not be written (they can, however, be read with the tlk2xml tool).

Additionally, the convert2da tool gained the ability to write binary 2DA files, as used by the two Knights of the Old Republic games; and xoreostex2tga can now correctly read TPC cube maps.

Binaries for Windows, GNU/Linux and Mac OS X are attached to the GitHub release, here for xoreos and here for xoreos-tools. Additionally, packages for various GNU/Linux distributions can be found on the OpenSuSE Build Service (here for xoreos, here for xoreos-tools) and in Arch Linux’s AUR (here for xoreos, here for xoreos-tools).

Alternatively, the repository and the source tarballs contain PKGBUILD files in dists/arch/ and a debian build directory in dists/debian/, which can be used to build Arch Linux and Debian/Ubuntu packages, respectively.

And as always, we’re looking for more developers to join us in our efforts to reimplement those 3D BioWare RPGs. If you would like to help, please feel free to contact us. :)

We are proud to announce our very first release of xoreos and xoreos-tools, version 0.0.2, nicknamed “Aribeth”.

While xoreos is still far from being useful to end-users, all targeted games work insofar as that they at least show basic in-game areas. You can start the game, xoreos loads the game resources, loads a campaign or module, and then shows an area of the game. This accurately demonstrates what the xoreos project wants to accomplish.

Neverwinter NightsNeverwinter Nights Knights of the Old RepublicKnights of the Old Republic Dragon Age: OriginsDragon Age: Origins

Within the in-game area, you can fly around in a “spectator” mode, using the common first-person WASD control scheme. Moving the mouse while holding down the middle mouse button rotates the camera. With Ctrl+D, a debug console drops down, allowing for general resource dumping and the loading of different areas, modules and/or campaigns.

A few games, specifically Neverwinter Nights and Knights of the Old Republic, also show a main menu, although the latter’s is not as extensive yet. The former also shows a few in-game menu elements.

Additionally, Neverwinter Nights also has a script system hooked up, and preliminary dialogue support. This means that clicking on an NPC opens up its conversation dialog, and some of the script commands will be executed. For example, the door in the first area of the original campaign’s prelude opens after speaking to Bim and telling him that no tutorial is necessary. However, triggering the tutorial leads to the scripts looping endlessly, because the necessary game functions are not implemented yet.

Further gameplay is still missing. At the moment, none of the other games have a script system.

The current graphics are very basic: only flat-shaded, textured meshes are shown. No lighting, shadows or shaders of any kind are currently available.

Please note that xoreos is still missing a GUI and needs to be started from the command line.

The accompanying xoreos-tools package includes command line tools that can be used to inspect the games’ resource files and, as such, are meant primarily for developers.

Binaries for Windows, GNU/Linux and Mac OS X are attached to the GitHub release, here for xoreos and here for xoreos-tools. Additionally, packages for various GNU/Linux distributions can be found on the OpenSuSE Build Service (here for xoreos, here for xoreos-tools) and in Arch Linux’s AUR (here for xoreos, here for xoreos-tools).

Alternatively, the repository and the source tarballs contain PKGBUILD files in dists/arch/ and a debian build directory in dists/debian/, which can be used to build Arch Linux and Debian/Ubuntu packages, respectively.

And as always, we’re looking for more developers to join us in our efforts to reimplement those 3D BioWare RPGs. If you would like to help, please feel free to contact us. :)

Yet further down the path of getting all targeted games to show areas, it seems like I reached the end with Dragon Age: Origins and Dragon Age II. Similar to my posts about my progress with Sonic Chronicles: The Dark Brotherhood (Part 1, Part 2, Part 3), The Witcher, Jade Empire and Neverwinter Nights 2, this will be a short description of what I did. This time: Dragon Age: Origins and Dragon Age II.

Models

Lucky for me, the Dragon Age model format is reasonably well documented in the Dragon Age toolset wiki. tazpn even created standalone model viewers for Dragon Age: Origins and Dragon Age II, and released them with sources under the terms of the 3-clause BSD license. :)

And since the model format is based on GFF4, missing pieces of information are relatively easy to decipher too. So I quickly had a loader capable of reading the skeleton whipped up for both Dragon Age: Origins and Dragon Age II (since they are nearly identical in format).

Rage demon skeletonRage demon skeleton Dragon skeletonDragon skeleton

With a bit of fiddling, the meshes were there too. There’s two types of meshes within the models: static meshes, directly hanging at one specific bone, and dynamic meshes that include weights for several bones for each vertex. Similar to models in Sonic Chronicles, this would deform the mesh according to those weights when the bones are animated. Unlike Sonic Chronicles, the default vertex positions of those meshes create a valid, unanimated pose. This means I could just completely ignore the bone weights for now, and load the meshes as if they were static. In the future, a vertex shader would combine those weights with the bone position to create the fully animatable model meshes.

Rage demon meshRage demon mesh Dragon meshDragon mesh Statue meshStatue mesh

Only thing missing now were the textures. For that, I needed to read the MAO (material object) files, which contains the material file (MAT), various textures (diffuse, lightmap, etc.) and a number of optional parameters. The material file in turn contains several different “semantics”, which is basically the name of a shader and how to map the MAO values onto the shader input. The original game takes all these, looks for the most fitting semantic in the material file (depending on number of parameters, graphics card capability and user settings), and then tells the graphics card which shader to use to render the mesh.

Now, since we don’t actually support any shaders yet (and we can’t use the game’s Direct3D shaders directly anyway), we simple read the MAO (which can be either in GFF4 or XML format), take the diffuse texture, and apply it to the mesh directly.

Textured rage demonTextured rage demon Textured dragonTextured dragon Textured statueTextured statue

Campaigns

With the models done, I turned to reading the Dragon Age: Origins campaign files. A campaign, that is either the default single player campaign (which is defined in a CIF file), or a DLC package (with both a CIF file and a manifest.xml) that doesn’t extend another campaign (those would be add-ons).

There’s several caveats involved here:

First of all, most of the DLC packages are encrypted. The original game queries a BioWare server for the decryption key, asking whether its a legitimate copy. While the encryption method is known (Blowfish in ECB mode), xoreos does not include any of the keys. So the only campaigns apart from the main one loadable right now are the unencrypted ones, namely Dragon Age: Awakening, and any custom ones you might have downloaded (including the PC Gamer promo DLC A Tale of Orzammar).

Then, we don’t load any add-ons. So no Shale or Feastday Gifts, even if they weren’t encrypted (which they are). It’s not like xoreos could do anything with them yet anyway.

Finally, we have no way to install .dazip packages yet, so those need to be installed using the original game for now, or manually extracted and put in the right places. In the future, something that install them would be nice. Or maybe we could support loading of packed .dazip files, but that could be slow.

In either case, I implemented the loading of standalone campaign files.

Areas and rooms

Next up were areas (ARE) and environment layouts (ARL) with room definitions (RML). The ARE contains dynamic room information, like what music to play, and the placeables and creatures (more of those later). The ARL defines what rooms are in the area (as well as pathing information, weather, fog, etc.), each of them being a RML file with models. They are all, again, GFF4 files, making them nice and easy to understand.

ArenaArena CastleCastle OstagarOstagar

There was one problem, though. The orientations of the models were given in quaternions, and as I said in the blog post about my The Witcher progress, a combination the automatic world rotation xoreos does, and our Model class wanting Euler angles instead leads to them not being correctly evaluated for whole models.

I was getting sick of that not being correct. I bit the bullet and removed the world rotation (which meant I had to rejigger the placement code in all engines, as well as the camera system, which was especially painful in Sonic Chronicles). And then I changed the Model class to take axis-angle rotations instead; those can be more easily calculated from quaternions, and can still be directly fed into OpenGL.

As a result, the area room models in Dragon Age: Origins were correctly oriented. And the placeable models in The Witcher as well.

Elven alienageElven alienage OstagarOstagar

You might notice that the ground mesh in outdoor areas looks very blurry and low-res. That’s because the original game doesn’t specify a single texture for those, but instead combines several textures together in a shader. We don’t support that yet, so instead we apply the replacement texture of the lowest LOD which is normally used for meshes that are far away.

Placeables

On to the placeables, the objects within areas. They are defined within a list in the ARE file (giving position, orientation, name, etc.), each with a template. The template is a UTP file, a GFF3, that contains common properties for all instances of this placeable. This includes an appearance, which is an index into a GDA (a GFF’d 2DA, a two-dimensional table), which specifies, among other things, the model to use.

So far, so usual for BioWare games.

One difference, though. In the Dragon Age games, the GDA files do not stand alone. Instead, each is a combination of potentially several GDA files with the same prefix (defined in m2da.gda). This is used for DLCs, which then can simply add rows to a GDA, instead of overwriting the whole file. Consequentially, the appearance index is not a direct row number, but corresponds to a value in the “ID” column.

A bit fiddly, but still relatively easy to implement.

PlaceablePlaceable PlaceablePlaceable

Creatures

The creatures were more difficult. There’s several types of creatures: type S (simple) are just a single model; type H (head) are split into a body model and several models for the head (base, eyes, hair, beard); type W (welded) are similar to H, but already include weapons in the body model; and “P” (player-type) creatures are segmented into head (with base, eyes, hair, beard), chest, hands (gloves) and feet (boots).

HurlockHurlock Headless DuncanHeadless Duncan Headless kitchen boyHeadless kitchen boy

Moreover, creatures of type P also switch model parts depending on the equipped items. So armor changes the chest model, gloves and boots change the hands/feet models and a helmet replaces the hair. Which models to use depends on several factors, and includes look-ups in several different GDA files, as well as UTC (creature template) and UTI (items) files.

Another problem is the tinting. The original game uses a shader to tint hair, skin and armor parts custom, user-selectable colors. To do that, their textures just contain intensity values in two color channels, while the two other channels are used as a bump map and something else (which I’m not sure yet). If we just apply the texture to those body parts, they are suddenly mostly transparent. To work around that for now, we manually modify each of those textures to remove the transparency. That leaves the weird coloring, but you can at least see all the body parts then.

Duncan without hairDuncan without hair Kitchen boyKitchen boy CookCook

Meeting of the headsMeeting of the heads BodiesBodies HelmetsHelmets

Dragon Age II

I then applied all this to Dragon Age II. Just a few minor changes to the resource loading was necessary, and nearly everything worked out of the box.

Hawke EstateHawke Estate ViscountViscount’s Keep

Only the P-type creatures needed a bit more work, since how the body part models are constructed changed.

CompanionsCompanions

Similar to Sonic Chronicles, Dragon Age II is also missing many of the GDA headers; they’re only stored as CRC hashes. With a dictionary attack, I did manage to crack about half of them, but that still leaves about 450 unknown. Something to watch out for in the future.

Music

I also investigated how music works in the two games. Dragon Age: Origins uses FMOD, and Dragon Age II uses Wwise. Both work similarily: the area specifies an event group, and the scripts then tell the library to play a specific event list from that group at certain times. The library does the rest, evaluating the events in the event list (which range from “play sound X”, over “set volume to Y”, to “add Z% reverb”). And while I do have adequately licensed code to read the sounds from both libraries’ soundbanks, figuring out the events is a massive undertaking. And we don’t have a script system for the Dragon Age games in place anyway, so this is nothing that can be done right now.

What’s next

So… All games xoreos cares about now show areas. What’s next, then?

Well, first of all, I’d like to do some cleanup of the engines code. Sync them up, make them more similar to each other. Right now, many things are done slightly different in each engine, because the games changes something around and the old concept suddenly didn’t fit anymore. If possible, I’d like to unify the concepts again.

There’s also a few potential portability issues I want to investigate. For example, I read that using fopen() on Windows with filenames containing non-ASCII characters won’t work at all. Instead, I’ll probably have to change xoreos’ File stream class to use Boost’s fstreams, and convert our UTF-8 strings to UTF-16 on file open. I hope that’s something I can test with Wine, otherwise I’ll have to bug somebody with access to a real Windows.

After those things have been cleared, I’d like to prepare for our very first release. I plan to include both xoreos and xoreos-tools, with sources (of course) and pre-compiled binaries for GNU/Linux, Mac OS X (>= 10.5) and Windows, each for both x86 and x86_64. I have cross-compilers for those, and they all should work. Yes, xoreos is still not really useful for end-users, but a release can’t hurt, and might give us some publicity and/or get people interested. Who knows.

I could use some testers for those binaries, though, to make sure I get the library dependencies correctly. And that the GNU/Linux binaries work on other systems than just mine.

I’m also open for other platforms. Would it make sense to have xoreos pre-compiled for Free/Net/OpenBSD? Other architectures than just x86/x86_64? Anybody with insights there, and capable of compiling those binaries (or pointers to cross-compilers), please, contact us. :)

As for how to continue the actual xoreos development, I think it would be useful to transfer the script system that’s currently hooked up to Neverwinter Nights onto the other engines. It would need to be rewritten, though. When I first wrote it, I wanted to have engine functions with signatures that mirrored the signatures of what the scripts call. I couldn’t get it to work, though, and settled on a context that contained an array of parameters. For some reason, I still used boost::bind for all the functions, which, at that point, was not necessary. boost::bind compiles really, really slow, and so now the files containing the Neverwinter Nights engines functions take ages to compile. This needs to go.

There, that’s the current short-term roadmap for me: cleanup, release, script system.