Arno Gerretsen : Visual databases

COLLADA reader

arno — Sat, 21 Nov 2009 11:37:00 GMT

Last I week I gave Google Sketchup another try and it seems a program you can learn to use quite quickly. If you compare it with GMax I think it has a learning curve that is much less steep. And I think if you model carefully in Sketchup you can make models with it that are suitable for real time rendering. So I think that for some people who just want to make some simple objects Sketchup might be a nice alternative.

But how to get those objects into FS? The free version of Sketchup only exports KMZ and DAE files. You could probably convert them to 3DS, import into GMax and then export, but I think all those extra steps make things too complicated. So I decided to work on a ModelConverterX feature that has been on my list for a while already, support for COLLADA DAE files.

I am not there yet, but I can now read the basic geometry from COLLADA files into ModelConverterX, below you see a screenshot where I loaded the COLLADA test duck. I still need to work on reading the materials used correctly and I guess some testing to see how robust the code is would be nice as well. But with a bit of luck I will have a beta of the COLLADA reader in the development release soon.

LOD Creator improvements

arno — Sun, 18 Oct 2009 11:29:00 GMT

The functionality in ModelConverterX to create lower level of detail versions of a model was running very slow for complex objects. I have not fixed that completely (yet), but I have made some changes to this functionality that should at least make it easier to work with. The calculation that simplifies the object is now running on a different thread, so that at least the rest of the tool should remain more responsive. I have also added a progress bar, so that you can see how far in the simplification process the tool is already. See the screenshot of the new dialog below.

As you might see there are some more changes. I have removed the 3D preview from the LOD Creator dialog. Instead the mean 3D preview is used now to display the result of the LOD Creator dialog. You can still select the LOD you want to see from the LOD Creator dialog though.

Optimizing the simplication code further, so that it will run faster is still on my todo list. But at the moment I am also busy with preparing for the new gPoly ground polygon tool. So I am not sure which one will get my attention first. Now that I am talking about the gPoly tool, I would still be happy to receive your input on what you would like to see in a tool for custom ground polygons.

3DS exporter improved

arno — Sat, 03 Oct 2009 09:20:00 GMT

I have just fixed some bugs in the 3DS exporter of ModelConverterX. The main issue solved is that complex objects often could not be imported into GMax. Without going into too much technical details, this was caused by some variables being stored as a short, allowing a maximum value of 65535. Some object parts had more triangles or vertices than this limit and that resulted in the error when importing. This has now be been fixed and those model parts will be split when exporting.

Another improvement I made to the 3DS exporter is that the night textures and bump textures are now also written to the material when exporting.

For future improvement of the 3DS exporter I still have the wish to add animations as well, but that is a little lower on the priority list at the moment. And another 3DS related wish is to have an importer for the 3DS format as well.

QGIS

arno — Sun, 15 Mar 2009 18:05:00 GMT

Until now I was mainly using fGIS when I wanted to view or edit shapefiles, but since a year or two the updates to fGIS are no longer publicly available due to some license issue. Last week I bought an interesting book about opensource GIS and it brought the QGIS application to my attention again. I had tried an earlier build of this application some time ago, but then I found it more difficult to use than fGIS. Now I tried a more recent build and I am impressed, it is a really powerfull application and quite easy to use. So I will replace fGIS by QGIS as my default GIS application for viewing and editing vector data.

Wavefront OBJ

arno — Wed, 04 Mar 2009 21:08:00 GMT

One of the new features I have started working on since the release of ModelConverterX 1.0 is adding support for the Wavefront OBJ format. One of the reasons for this is that we use it at work for some files and I would like to try the LOD Creator functionality on those. But I think it could also be an alternative to the 3DS format for some people who try to import their old work into other tools. The 3DS format has some annoying limitations, like texture names having the 8.3 DOS naming convention. At the moment it seems to me that the OBJ format has less of such limitations.

In the current development build of ModelConverterX the OBJ reader is already implemented and until now it seems to work well on all test objects I have found until now. At the moment I am programming the OBJ writer, so this should be available for testing soon as well.

At first the writer will only support the object geometry, I will have to see if I can implement advanced features like animations or attachpoints. I will do some more study on the format for that later.

ModelConverterX 1.0 release

arno — Fri, 27 Feb 2009 13:02:00 GMT

I don't want to repeat myself here again, so I will just point at this forum thread where I explain all details of the ModelConverterX 1.0 release.

Why is the 3DS format giving me trouble?

arno — Mon, 24 Nov 2008 18:32:00 GMT

Over the weekend I have been continuing with the ModelConverterX tool again. A while ago I improved the FSX MDL importer so that it could read levels of details and animations, but at that moment I did not update the different exporters to handle these more advanced features as well. So now I decided it was a good moment to address these issues. I should mention right away that getting animations to work in the exporter is a step that I did not try to take yet, now I focussed on getting the static transformations and the levels of detail right.

For the X file exporter, which is used to make FS2004 and FSX MDL files, it was relatively easy to add the support for these features. And also for the OpenFlight exporter it did not give me too much trouble. I was actually quite pleased to see that my translation from the FSX level of detail numbers to distances, as used in the OpenFlight file format, worked out very well. After I loaded the object with the OSG viewer, the transition between the different levels of detail was very smooth. So that indicates the distances were choosen quite well.

But for the 3DS file format things were not as easy unfortunately. The more I look at that format, the more I start to wonder why it is used so much to interchange models between different formats. I know the answer in this case is that 3DS is the only format GMax can import, so that is why I am bothering myself with trying to implement it. But the format has some quite annoying drawbacks. The texture names in the materials can only be in the old 8.3 format from the DOS days for example and not so many people seem to restrict themself to using only eight characters at the moment. Another annoying one is that the part names can only be 10 characters. Giving the fact that FS uses a LOD naming schema like _LOD_100 you can do the math yourself to figure out how much characters are left to name the part apart from the LOD part.

But the good news in the end is that I also got the 3DS exporter to work relatively well, considering all these limitations. Now the next item on my todo list is to look at the code to automatically generate lower detail version of your model for the levels of detail.

ModelConverterX update avaible

arno — Fri, 30 May 2008 17:25:00 GMT

Today I handed in the key of my old house, so the moving "project" is now really finished. The last days I already had some more time to do some programming on my FS tools again, so as a result of that I have also been able to release a new version of ModelConverterX.

On the forum you can read all the changes in this new version, but the biggest new feature has to be the 3DS export capability. That opens new possibilities again to bring your old objects to life again in FSX. I think the next challenge I am going to tackle is making LOD models with the tool and writing them out as well.

ModelConverterX 3DS exporter

arno — Sun, 18 May 2008 07:25:00 GMT

Now that everything is settled a bit again in the new house, I finally found some time again to do some programming. So the last evening I tried to continue on the 3DS exporter for ModelConverterX, that I had been working on before the moving. And the good news is that I was able to fix the last remaining bugs in it. So the basic exporter functionality is working now. See the screenshot below for the museum API that I use often to test ModelConverterX, here it has been exported as 3DS and then loaded into GMax.

Currently the exporter only deals with the geometry, so no animations or advanced features like that are written (besides from the fact that ModelConverterX can not even read these features yet, the exporter does also not support them at the moment). An exporter like this opens a whole new world of reuse of your old scenery objects, like API macros.

But there is also one possible problem with it, it does allow people to import and edit work that is not their own. In general developers do not like that, so I think I will have to consider how to release this functionality in such a way it is useful, but restricts people from doing "bad" things with. If anybody has an idea about that, please let me know as well.

Real-time rendering

arno — Sun, 24 Feb 2008 16:06:00 GMT

Recently I bought an interesting book, Real-time rendering. I have not completely finished reading this book, but it contains a lot of useful information. For example I have found there solutions for some problems I had while working on my ModelConverterX tool. But the explanation of how the rendering process works also give a better insight on what has an influence on the performance when rendering. This is not only interesting to understand better how FSX works, but that is also something I can use at work when dealing with our image generators there.

SimConnect and CIGI

arno — Wed, 16 Jan 2008 21:35:00 GMT

For some while already I wanted to learn a bit more about SimConnect and this week I found a good reason to do so. I wanted to try if I could connect FSX to CIGI. CIGI stands for Common Image Generator Interface and this is a common way in professional simulations to talk between the simulation host and the image generators. At work we also make use of CIGI for this.

What I wanted to do is try to use FSX as an image generator with our simulation host, so for that I needed to connect FSX to CIGI. This is where SimConnect was very useful, after some hours of trying I am now able to control the user aircraft with the CIGI interface and I can also add other traffic (AI) and control them from CIGI as well. This is a nice demonstration that the concept works.

Until now I only tested it with the host emulator that CIGI delivers, so a next step will be to attach it to the real simulation at work. Should be interesting. Of course I am not ready yet, as other things like the weather should still be connected as well.

I guess such a CIGI connection might also be very useful for ESP, it would open a lot more possibilities in the professional flight simulation world. Sounds like great fun to play with that some more at a later time...

OpenFlight exporter for ModelConverterX

arno — Fri, 28 Dec 2007 16:26:00 GMT

I can remove the first item from my "what I would like to do during the Christmas vacation" list. Today I finished the OpenFlight exporter functionality of the ModelConverterX tool. With this functionality it is possible to save the loaded objects in the OpenFlight format that is often used in image generators of professional flight simulators. The object shown in the picture below is an old API macro of a museum that I exported to OpenFlight and then loaded into OpenSceneGraph to visualize it.

The exporter is not yet perfect of course, it only supports static objects at the moment. And also levels of details are not yet supported. But that are things on my wish list and now that the basic functionality is there it is a lot easier to add new things to it.

I have also made progress on improving the SCASM reader in the last days. Especially the part that processing the old SCASM Poly and TexPoly commands has been improved a lot, so that more macros can be read correctly now. There are still some minor problems I need to fix, but I feel I am getting closer to a next alpha version that will be released to the public.

Merry Christmas!

arno — Sun, 23 Dec 2007 13:11:00 GMT

Christmas is almost there, so I would like to take this opertunity to wish you all a Merry Christmas.

Of course I will spend the Christmas days with my family, but for the rest this time of year is a good opportunity to catch up with some of the things I want to do for a long time already. So those two weeks without having to go to work are already quite full with plans for things to do. I guess the best thing I could wish from Santa Claus (apart from the fact we already celebrated Sinterklaas, so no Santa for me) is some more time to spend on all this. Here is just an overview of some of the things I have in mind, don't pin me down on this as I will probably not be able to finish it all:

Finish a new version of ModelConverterX and release it. Over the last week I have been running a test project to convert the objects from a certain airport and that has resulted in a lot of improvements to the SCASM reader functionality of the tool. Once I can convert all objects I want to convert, I will release a new version.
Add OpenFlight funtionality to the ModelConverterX tool. With my MDL Tweaker II tool I could already write out OpenFlight versions of my MDL objects. I want to add the same functionality to ModelConverterX. That would allow me to convert SCASM objects to OpenFlight and later also MDL objects when I can read them in again. And even more interesting I am also thinking about an OpenFlight reader, so that you can put OpenFlight models into FSX MDL objects.
I want to study the SimConnect SDK a bit more, to get a better understanding of what can and cannot be done with it. As part of this I will probably try to write a little tool to drive FSX with external data (that could be useful later at work if we want to use FSX as a visual for our simulations).
I want to look into the conditional display of scenery objects. At the DevCon in November I got some interesting suggestions that might allow (some) conditions to work again. As the MDL format does not allow it directly, most suggestions involve using Missions and/or SimConnect. So that will be two fields I will be diving into a bit.

As you can see the list is quite long, let's see how far I can come. At least it seems I do not have to be bored during my Christmas holidays...

Microsoft ESP

arno — Mon, 03 Dec 2007 13:57:00 GMT

I know this is old new, and I guess you must have read about this multiple times already by now. But as I think it is still very interesting I am writing this blog post about ESP. ESP is a new simulation platform that Microsoft has recently announced and it is based on the same simulation engine as FSX. Or to quote the description from Microsoft themself:

"Microsoft ESP is a visual simulation platform that brings powerful, high-fidelity simulations to widely available, low-cost Microsoft Windows®-based PCs. Organizations that rely on Microsoft ESP can improve their workforce readiness more efficiently and cost-effectively than with traditional computer-based animation or simulation tools. Combined with its developer-friendly Software Development Kit (SDK), Microsoft ESP provides government and commercial entities and modeling and simulation specialists with an affordable, portable, and highly extensible platform for training and learning, decision support, and research and development modeling.

Simulations built on Microsoft ESP can engage users in immersive experiences with very realistic land, sea, and air environments—making them ideal tools for training. With a high-speed Internet connection, up to 30 users, including students, instructors, and support personnel, can participate simultaneously in Microsoft ESP simulations from any location in the world. In addition, the platform’s built-in analysis and full playback capabilities facilitate evaluations of crew performance, mission planning, and decision-making."

I find this a very interesting product, as the use of the FSX engine can also have great benefits for non-entertainment use. For example at work we sometimes already use FS2004 as a quick visual for our simulations, simply because it covers the entire world by default. That is not something that can be said about any image generator. This can save us a lot of time in having to design or update visual database to include a certain area. And also with the FS SDK tools it is possible to add new data faster to FS than for most other image generators. And another interesting reason to use FS is that because it has been designed for entertainment usage, the resulting graphical image is a lot better in general.

So the fact that Microsoft has now announced a professional version of FS sounds very interesting, given the benefits I mentioned above. I have also written before about my efforts to convert between FS and OpenFlight (the format used in most other image generators). Now with ESP I see a lot more connections between the FS and the "professional" world and I am certainly looking forward to explore those new options. So hopefully I can report some more interesting things later on.

Day 5: DevCon 1st day

arno — Thu, 08 Nov 2007 05:14:00 GMT

So today was the first day of the DevCon, my main purpose of this travelling. On this blog I can not talk about all the actual stuff being discussed there, as we are under NDA for that. But I certainly did meet a lot of interesting people. I am sure I will forget to mention somebody here, so sorry if I forgot you.

For example I met Bob Bernstein and Bill Womack, two people I know for years already from different internet fora for scenery designers. It was really nice to talk with them in person. I also met different users of the FSDeveloper website, luckily most recognised me from my avatar photo. I also met with Russell, the author of FSX Planner and we had some very interesting talk about his tool and scenery development in general.

And of course I should not forget to mention all the people from the ACES team I talked with during they day. Being able to talk so easily with them about things that we (as the community) think are important is a great benefit of an event like this.

I also gave my presentation today, or maybe I should say presentations as I had two different subjects to talk on. The first part of my presentation was about the FSDeveloper community and how we try to support other addon developers with the forum, wiki, etc. The second part of the presentation dealt with the OpenFlight conversions I have worked on in the past. Or maybe that is even more an example, my talk was more about the possibilities for conversion in general.

COLLADA

arno — Mon, 07 May 2007 19:33:00 GMT

For the new model conversion tool I am working on at the moment, I took a closer look at the COLLADA format this weekend. COLLADA is an open standard Digital Asset schema for interactive 3D applications. It can for example be used as an exchange format to share models between different tools. That is also the role in which I want to use it in my tool. There are for example COLLADA plugins for tools like 3DSMax and Maya, but also Blender can work with it.

I think that it would be useful to have the ability to convert COLLADA files to FsX MDL files, so that more models can be used with FsX. And also the reverse conversion might be useful if that allows you to edit your models easier (but I have not looked deep into that yet).

So a COLLADA importer/exporter is certainly on the todo list for my new tool. But it does not have the highest priority at the moment. The first features I want to have working is importing old API macros and being able to save them again as MDL obect. As that is the main purpose of this tool. After that I'll see what other cool features can be added to it.

Having fun with projections

arno — Wed, 14 Feb 2007 17:40:00 GMT

Using GIS data to create your scenery has become a lot more common in FsX. Vector data can for example converted from shapefiles and for new version of resample can process GeoTIFF images as well. If you have access to this kind of GIS data or can create it yourself, this makes creating the scenery a lot easier (and more fun as well).

But there are also aspects of working with GIS data that are hard to get for people not used to this. One of these is the use of projections. All FsX scenery design tools expect the data in the WGS84 projection. This means that you can usually not use any scanned map.

Today at work we came across a similar problem with the use of projections and I want to talk about it here as well, because it illustrates very nicely what a big influence a wrong projection can have.

For the flight simulator we have a visual database that contains the airport of Paris CDG and Sion, plus some terrain between these two airports. So the projection used for this database has been optimized for the area it covers, we are using a Lambert Conformal Conic (LCC) projection for this database. A few days ago they asked us if we can add an airport in Africa (Mali to be more specific) as well, because they wanted to fly from there to Paris for an experiment.

So our first thought was let's try to add the additional airport to the same database (which also means using the same projection). So we downloaded some Landsat images of Mali and started adding them to the database. But to our suprise (or maybe not really), the results were not very good. Below you can see a screenshot of the airport in the LCC projection of our Paris database (left) and a screenshot of the same airport when we use a UTM projection chosen for Mali (right).

As you can see, the heading of the runway differs something like 60 degrees. Which is of course much to much (how to explain the pilot that runway 06-24 is actually facing North). To be honest we already expect some trouble with the projection system, but we just want to try and see how worse it was. And it was much worse than we ever expected.

But I think this nicely shows that you can not ignore the projection you are using when working with GIS data. To prevent problems like the one we had, MS has chosen to use WGS84 for all scenery elements. So to get the correct placement, all you have to do as a scenery design is to make sure your data is in WGS84 or else convert it.

Am I a data junkie?

arno — Fri, 19 Jan 2007 21:09:00 GMT

At work my collegue and I, we are the ones working on the visual databases for the simulators, are quite well known for our excessive harddisk usage on our development systems. When we are using satelite images in these visual database, it is not that weird to have a few GB of images that we need to process. For example to cover the Balkan area or the entire country of the Netherlands. And then I am not even talking about high resolution images (that would increase the amount of diskspace used even more).

During the processing of these images into the actual databsae, we usually have to reproject them a little or apply some other tweaks on them and of course the final tool creating the database also generates some temp files worth a few more GB of data. So I guess you can imagine that we have to clean our harddisk quite often to keep the system running (from experience I know that 0 kB left of harddisk space does not really work well). And I guess the system administrator of our department has already gotten used to our questions for more diskspace.

And now the same seems the be happening at home with FsX. With the increased resolution of the terrain system in FsX the amount of data to process also goes up of course. I did a little test with high resolution images around Schiphol airport (16 cm resolution). Those photo alone were already a few GB worth of data, but when I added some intermediate files due to the conversion from the Dutch RD coordinate system to WGS84 and mosaicing of the different images into a few larger ones, the harddisk usage for this little test went up already. Now, after compiling the BGL files for FsX as well, the entire test project uses about 50 GB of diskspace (of which the final BGL files are only about 6 GB). Luckily my new PC at home has a big enough harddisk (for the moment).

With the changes to the FsX terrain system, I find it interesting to see that the process of scenery creation comes a step closer to the GIS data I have become familiar with at work already. The new shp2vec tool makes use of shapefiles and resample can process GeoTIFFs. Quite interesting developments and it shows that the difference between "professional" visual database and the FsX "entertainment" world is not that big.

Texture and material blend

arno — Wed, 15 Feb 2006 21:45:00 GMT

At work we have recently updated our image generation software and the models that I had converted from GMax models suddenly looked a lot darker in the visual. To dark actually. So after a bit of searching we found out that the new version of the IG blends the material and texture, while the old on did not. So because the ambient color of the material was not white, but gray, it resulted in a darkening of the texture. With this extra knowledge that was easy to fix of course.

Because I was interested to see how FS behaves here, I did a few tests with blending of material and texture there as well. I posted about the results on my forum, but the main conclusion is that FS only blends the emissive color. All other colors do not have an effect on the display of the texture. Only the combination of a specular color and the power parameter seems to have some effect, but I can not create a useful visual effect with them. It only seems to distort the image.

OpenFlight

arno — Tue, 13 Dec 2005 22:11:00 GMT

I want to start with a little warning. I am afraid this post is going to be rather long, so if you are already desperately in need of coffee now, you'd better get a cup before reading the rest of this.

So what is this OpenFlight you are talking about you might think? OpenFlight is not some sort of open source Flight Simulator or so. No it is a very common format to store your scenery in, for professional flight simulators. In this article I want to talk about the similarities between this format and the BGL format of Flight Simulator. And also about my attempts to make a converter between the two formats. Let's get started.

The OpenFlight format has been create by MultiGen-Paradigm and is very common in real flightsimulators. At work we also make use of this format to store our visual databases (or scenery as most of you will know it). Rather soon after I started on my job, I had to learn more about the OpenFlight format, as I wanted to create a few tools to make our work easier. As I can almost dream the FS BGL/MDL format by now, it was rather logical that I compared this new format for me to what I knew. Slightly to my surprise a lot of things are almost the same. All materials are defined with the same properties (diffuse, ambient, specular and emissive colors for example) and the way to to store vertices and create triangles using them is also very similar.

The biggest difference for me was, that the OpenFlight format is a sort of a database. So each polygon has a few vertex nodes as children. And as a parent the polygon node can for example have a group, object or LOD node. This makes the structure of your object a lot clearer. A downside of this, is that the file format itself takes more diskspace. While learning about OpenFlight, I started to understand that MS has chosen a nice compact format to store the objects in.

There is another difference I should note. When I am talking about FS BGL/MDL files, I am talking about the object files mainly. The terrain has its own BGL format in FS, which has its own logic. In the OpenFlight format we usually create an entire database in one file (or a set of files that are linked together). So you have terrain, roads, 3D objects, airport all defined in the same file. As a result of this, the structure of your file is much more important, when trying to optimize your scenery. In FS scenery the MDL files usually contain only one object, so that makes their structure less important. All the placement is done with different (XML based) code and that has its own logic and structure again (I won't start to talk about that as well here).

At work we were working on a database for Schiphol, the airport of Amsterdam. And because I had already created a FS scenery for this airport (as part of the NL2000 scenery team), the question soon came if we could not convert some of those 3D objects for use in the real flight simulator. One of the other NL2000 members had already created a converter the could create OpenFlight files from SCASM source code. But as we had created these objects with Fs2004 MDL files, they could not be decompiled to SCASM code. I had also started with the new version of my MDL Tweaker tool (which contains a nice MDL loader) at that moment, I decided that it would be cool if I could also add an OpenFlight export function to it.

In theory this conversion is not that hard. You have an object, that contains of a collection of vertices, materials, textures and triangles and you need to read the MDL format and write the geometry to the OpenFlight format. And the logic behind the conversion was indeed finished rather soon. But then the trouble started for me. I had to code this logic. The API provided with the OpenFlight format provides functions to create an OpenFlight file, but unfortunately for me these are in C++. And because I still need to find an easy way to create a nice GUI in C++, I am coding my MDL Tweaker project in Visual Basic.

So I first tried to use the functions of this API in Visual Basic. I won't bother you with all technical details, but it proved to be more difficult then I had expected. Either the export function worked correctly, but crashed on big MDL files, or it worked on big files, but the properties in the OpenFlight format were not set correctly. So a few days ago, I decided that it would be easier if I wrote directly to the binary OpenFlight format from MDL Tweaker. A good discription of this binary format is luckily available.

So just this evening, I was able to finish that part of MDL Tweaker and the good news is that now everything works fine. So I can create OpenFlight files from my MDL scenery objects created with GMax. Now you also know the reason for the delay of a new alpha version of MDL Tweaker. The last few weeks I have mainly been busy to implement this new function (I could not accept that it did not work as I wanted).

After reading all this, you might think what can I do with all this? The answer for most of you is probably that the OpenFlight format is not of very much use to you. But if you want to play with it a little bit, I would advice you to give OpenSceneGraph a try. The OSGViewer that comes with OpenSceneGraph is able to load OpenFlight (FLT) files. So after I have released the next alpha version of MDL Tweaker, you can view your own objects in the OpenFlight format is you ever wish to do so :).