PasJpeg2000 Update in Progress

I’m currently rewriting much of a Jpeg 2000 for Pascal library. There is a new IO class responsible for decoding and encoding of Jpeg 2000 files instead of only VCL TBitmap descendant. It’s cross platform and with only Delphi/FPC RTL dependencies. VCL and LCL TGraphic classes will be built using this IO class but it can be used independently as well (Imaging library will use it too).

New features of Jpeg 2000 for Pascal will be CMYK colorspace support and also indexed/palettized images support (yes, it’s possible to have image using palette in Jpeg 2000). These features as well as proper alpha channel definitions are patched into OpenJpeg library. Its team is not very active in incorporating larger patches into their code, so patches will probably always be additional step for people who want to recompile OpenJpeg themselves for use with PasJpeg2000.

You can follow the progress of the new version in SVN repository here: branch v120.

Ugly Images of Disabled Menu Items in Delphi

Ever used 32bit images stored in TImageList in your Delphi application? Toolbars and some other VCL controls have DisabledImages property which is automatically used to get images for disabled toolbar buttons. But what about menu components? They don’t have this property and drawing of disabled images is handled by TImageList with original enabled images (TMainMenu.Images property). And the results are really abysmal. How can this be fixed?

One way is to override DoDraw method of TImageList and change the code that draws disabled images. You can do regular RGB to grayscale conversion here or let  Windows draw it for you in grayscale with nearly no work on your part. You can do this by using ImageList_DrawIndirect with ILS_SATURATE parameter. Note that this works only on Windows XP and newer and for 32bit images only. For older targets or color depths doing your own RGB->grayscale conversion is an option (good idea would probably be to cache converted grayscale images somewhere so they won’t need to be converted on every draw call).

Here’s the code of DoDraw method using  ILS_SATURATE:

type
// Descendant of regular TImageList
TSIImageList = class(TImageList)
protected
  procedure DoDraw(Index: Integer; Canvas: TCanvas; X, Y: Integer;
    Style: Cardinal; Enabled: Boolean = True); override;
end;

procedure TSIImageList.DoDraw(Index: Integer; Canvas: TCanvas; X, Y: Integer;
  Style: Cardinal; Enabled: Boolean);
var
  Options: TImageListDrawParams;

  function GetRGBColor(Value: TColor): Cardinal;
  begin
    Result := ColorToRGB(Value);
    case Result of
      clNone: Result := CLR_NONE;
      clDefault: Result := CLR_DEFAULT;
    end;
  end;

begin
  if Enabled or (ColorDepth <> cd32Bit) then
    inherited
  else if HandleAllocated then
  begin
    FillChar(Options, SizeOf(Options), 0);
    Options.cbSize := SizeOf(Options);
    Options.himl := Self.Handle;
    Options.i := Index;
    Options.hdcDst := Canvas.Handle;
    Options.x := X;
    Options.y := Y;
    Options.cx := 0;
    Options.cy := 0;
    Options.xBitmap := 0;
    Options.yBitmap := 0;
    Options.rgbBk := GetRGBColor(BkColor);
    Options.rgbFg := GetRGBColor(BlendColor);
    Options.fStyle := Style;
    Options.fState := ILS_SATURATE; // Grayscale for 32bit images

    ImageList_DrawIndirect(@Options);
  end;
end;

Important note: For ILS_SATURATE to work correctly source image files must be 32bit with proper alpha channel data, setting color depth of TImageList to 32bit is not enough! If you don’t see any images drawn this is probably the cause: 8/24bit image is loaded from file and then inserted into 32bit TImageList. As there is no alpha channel data in source image it is drawn as fully transparent so you don’t see anything.

http://msdn.microsoft.com/en-us/library/bb761537(VS.85).aspx

HoMaM 2 and Arena remakes

My favorite game from Heroes of Might and Magic series is the second one. I guess it’s mainly because of its graphics. I like these nicely done 2D sprites much more than renders of 3D models from HoMaM 3.

Today you can play original DOS version of  HoMaM 2 in DosBox. It wasn’t the case few years ago when DosBox wasn’t able to run SVGA games. There’s also Windows version of HoMaM 2 executables in Heroes Of Might And Magic 2 Gold (expansion disk included). However, neither of these two options take advantage of today’s high resolution monitors. Map view will always be stretched from it’s original 640×480 resolution.

Several years ago, I stumbled upon Free Heroes II project. It’s a GPL cross-platform (SDL based) implementation of Heroes of the Might and Magic II engine. It’s been steadily progressing over the years and fortunately hasn’t died like many other remakes of older games. Today, “standard game” mode is playable as well as hot seat multiplayer. Very nice feature of Free Heroes II is that map view is not stretched to higher resolutions. Instead, you can see larger portion of a map without the need for a lot of scrolling around. Let’s hope we’ll see version 1.0 in near future.

Free Heroes 2

Another game I’d like to see remade is the first The Elder Scrolls game Arena. There hasn’t been any known attempt at remake Arena – unlike its sequel Daggerfall (though all past remakes failed the latest one, DaggerXL, remains very promising). No description of Arena file formats is available on the Internet either (with few exceptions of raw images or formats shared by more Bethesda games) – again unlike Daggerfall where almost everything is described in detail.

I was pleasantly surprised few months ago when I found out about WinArena. It was originally Arena resource extractor and converter (textures, maps, sounds, sprites, …) with 3D engine capable of  displaying Arena maps added later. Unfortunately there are only two versions you can find on the net and the latest is dated December 2005 – so it’s long since dead.

3D engine uses software rendering in 256 color mode like original Arena (it supports higher resolutions though). According to the author, many of the effects in the game are based on palette manipulation tricks so it wouldn’t be possible to get the exact Arena look using standard hardware accelerated 3D API like Direct3D or OpenGL.

What’s best on WinArena is that full source code is provided. And since it handles most if not all Arena file formats, getting their description is just a matter of going through the code. I’d also love the possibility to revisit all the main maps in the game so maybe I’ll modify it a little bit (WinArena currently starts at the beginning of actual Arena gameplay – in the prison like most TES games do).

You can get latest version of WinArena at TESNexus.

WinArena

Just before publishing this post I also found Python port of Elder Scrolls: Arena which also seems inactive at the moment (there’s some resource extraction Python code in the project’s repository).

Stats for 2009

I quite like looking at various stats and sometimes even big tables with a lots of numbers. Since I don’t have anything in particular to write about now, I’ll share some of the site stats for 2009.

Downloads

Top download is JPEG 2000 for Pascal package with about 700 downloads. Second place goes to glSOAR terrain renderer with about 200 downloads. Third place is shared between Earth Under Fire shooter and venerable JPEG Ripper tool, both with meager 60 downloads.

Visitors

There have been around 4000 visitors since last March. 42% of them got here from referring sites,  35% used search engine, and 23% is direct traffic.

Referring Pages

Basically all of them are connected to Imaging Library or JPEG 2000 for Pascal (torry.net, delphiplus.org, imaginglib.sf.net, cc.embarcadero.com, etc.). For some reason there’s really a lot of referrals from images.google[.ru|.de|.fr|.com] pointing mostly to screenshots from Ufo: Extraterrestrials and Terrain Rendering stuff. Few links to Earth Under Fire from pascalgamedevelopment.com are in the stats too.

Browsers and OS

Firefox is in the lead with 40% share, IE is second with 25%, third is Opera with 22%, Chrome and Safari have 7% and 2% respectively. Windows is major OS here with 91%, Linux and Mac OS X have about 4% each. There’s also one visit from Wii and one from PS3.

Search Engine Keywords

Most common keywords refer to these topics: Terrain rendering, JPEG 2000 for Delphi and Pascal, Ufo: Extraterrestrials, DirectX 11 for Delphi, and Daggerfall. You can find more or less information about these topics at this site.

Some keywords were very common too even though I wouldn’t think they’d be so popular:

  • APNG  (mostly coupled with Delphi) – As far as I know my Imaging is only library for Pascal and Delphi that can read and write APNG animated images. That’s probably why Google search for “apng delphi” puts this site at top.
  • Delphi shr operator – Looks like there’s more people surprised that it treats negative values differently than shift right operator in C and some other languages. More on this in this post Shift Right: Delphi vs C.
  • Daggerfall music (“daggerfall gstory music”, “eric heberling” daggerfall”, “robert hood daggerfall”) – Great music indeed, much better than what’s in Morrowind and Oblivion.
  • SOAR terain rendering – here’s my Pascal implementation glSOAR.
  • Modula-2 related (“modula-2 vs c++, vs pascal, vs freepascal, on mac ox, for 8051, …” )

And here are some uniques searches:

  • “ancientlich” – My favorite monster in Daggerfall as well, checkout bestiary at UESP (top of the Undead section!).
  • “UFO old programming language” – that must be COBOL!
  • “ufo enemy ripped png picture” – anyone has these?
  • “directx 11 radeon x1950” – sadly no, otherwise I’d put it back in my PC right away.
  • “mng vs apng” – basically MNG is much more capable but harder to implement, more VS info here.
  • “john the ripper directx 11” – is this some of kind of a game or what?
  • “mac os 7.5 emulator” – check out Basilisk .
  • “midi format daggerfall” – It’s HMI (Human machine interfaces MIDI music file), WinAmp can play and convert that (and other old game MIDI formats as well).
  • “a price drop for gpu in 2010” – there better be one.
  • “earth on fire from space” – let’s hope not anytime soon.
  • “gulfar vivek”, “how is galfar doing?”, “large galfar image” – ???

Imaging in C++ Builder

I tried compiling Imaging in C++ Builder (it uses Delphi compiler to generate .obj file which C++ linker can link and also generates C++ header for Pascal unit) few years ago. It didn’t work – there was internal compiler error, I think right in ImagingTypes unit.

Few days ago I tried C++ Builder 2010 and was pleasantly surprised. It worked! I tried just the library core for now (ImagingTypes, Imaging, ImagingFormats, Pascal only file handlers, etc.) and it works without problems. I’m not sure which C++ Builder version is required for successful compile though. Versions 6 and 2006 stopped with internal error, 2010 worked, and there are 2 other versions between.

Anyway, I’ll try to check out most of the library until the 2010 trial expires for me. Hmm, I’m wondering how many people use C++ Builder for C++ development – I’ve never did something serious in it, basically just to get object files usable by Delphi – so I have no idea if it’s ok.

PS: Another C++ Builder related news – patch for OpenJpeg library to get it compiling in BCB is posted here.

PasJpeg2000 News

JPEG 2000 for Pascal project is based on OpenJpeg library. For a very long time there was a bug that caused alpha (fourth and subsequent image channels) channel to be saved with all the samples having the value of 0.5 (128 for 8bit channels). This buggy behavior also depended on compiler settings – optimization level in case of GCC. You could use at most O1 in Windows and Linux, and only O0 in Mac OS X. Bug was also present when compiling with C++ Builder (to get object files usable in Delphi) but only when irreversible DWT transformation was enabled in OpenJpeg during encoding (it wasn’t before, but working versions of both PasJpeg2000 and Imaging use it now when lossy compression is selected by user to get smaller files).
You can read more about it in this news group post. Basically it was all fixed by changing a condition in one if statement to prevent accessing the fourth element of a three element array.

So what can you expect in the next version of PasJpeg2000 library?
Higher GCC optimization levels should make it a lot faster when using Free Pascal (particularly in Mac OS X where O0 was used). Irreversible DWT transformation produces smaller lossy files than current PasJpeg2000 version and with optional MCT (multicomponent transform – basically RGB>YCbCr) you get even smaller ones. There’s now also a patch that enables OpenJpeg to get palettes from JP2 files so indexed JPEG 2000 images could be supported too. And finally, there are some bug fixes (wrong reconstruction of subsampled files, …).

Shift Right: Delphi vs C

Few weeks ago I converted a little function from C language to Delphi. I kept getting completely wrong results all the time even though I was sure the C to Pascal conversion was right (it was really just few lines). After some desperate time, I just tried replacing SHR operator by normal DIV (as A SHR 1 = A DIV 2 and so on). To my surprise, I immediately got the right results. Can Delphi’s (I didn’t test it in Free Pascal) SHR operator behave differently than C’s >> operator?

It does in fact. SHR treats its first operand as unsigned value even though it is a variable of signed type whereas >> takes the sign bit into account. In the function I converted the operand for right shift was often negative and Delphi’s SHR just ignored the value of the sign bit.

A Bit of Code

int a, b1, b2;
a = -512;
b1 = a >> 1;
b2 = a / 2;

After running this C code both b1 and b2 have a value of -256.

var A, B1, B2: Integer;
A := -512;
B1 := A shr 1;
B2 := A div 2;

This Delphi code however yields different result: B2 is -256 as expected but B1 has a value of 2147483392.

A Bit of Assembler

Assembler output of C code:

Unit1.cpp.22: b1 = a >> 1;
mov eax,[ebp-$0c]
sar eax,1
mov [ebp-$10],eax
Unit1.cpp.23: b2 = a / 2;
mov edx,[ebp-$0c]
sar edx,1
jns $00401bb9
adc edx,$00
mov [ebp-$14],edx

Assembler output of Delphi code:

Unit1.pas.371: B1 := A shr 1;
mov eax,[ebp-$0c]
shr eax,1
mov [ebp-$1c],eax
Unit1.pas.373: B2 := A div 2;
mov eax,[ebp-$0c]
sar eax,1
jns $00565315
adc eax,$00
mov [ebp-$20],eax

As you can see, asm output of C and Delphi divisions is identical. What differs is asm for shift right operator. Delphi uses shr instruction whereas C uses sar instruction. The difference: shr does logical shift and sar does arithmetic one.

The SHR instruction clears the most significant bit (see Figure 6-7 in the Intel Architecture Software Developer’s Manual, Volume 1); the SAR instruction sets or clears the most significant bit to correspond to the sign (most significant bit) of the original value in the destination operand.

Quoted from: http://faydoc.tripod.com/cpu/shr.htm

DirectX 11: GPUs and headers

Few days ago I was wondering how long it will take to see DirectX 11 headers translated to Delphi/Pascal. I knew that translations of new DX components Direct2D and DirectWrite are part of Delphi 2010, but what about Direct3D and DirectCompute?

I checked out Clootie graphics pages but no luck there. Unfortunately, it’s not so active anymore. After that I thought DX11 for Pascal wouldn’t appear in like several months at least. Surprise, I found it about 5 minutes later. John Bladen published his DX11 translation on DirectX for Delphi blog. Thanks John!

Now all that remains is buying a DX11 GPU. New Radeons should be released sometime during this week and I’m thinking of getting 5850 model in a few months. Rumours are that NVidia’s DX11 cards won’t be out before 2010, so probably no big price drops for 5800 Radeons are to be expected until then.

My pile of old graphic cards will get bigger once again. I don’t buy new CPU/motherboard/RAM too often (now it’s Core2 from 2007 and before that AthlonXP in 2002) and I usually manage to sell the old ones or put them on server duty somewhere. However, since GPU performance as well as requirements on it rise much more faster than that of CPU, I ended up with like 5 cards in last 8 years. I managed to sell Radeon 9000 and GeForce 6600 but I still have Riva TNT2 and Radeon X1950 in my closet somewhere (and S3 Trio!). Hopefully, I’ll find a buyer for my 1GB 4850 soon.

Lately, I’ve had bad luck buying some new stuff just before major price drop. Hopefully, this won’t happen with 5850. Not that I want it to remain expensive – just want to buy it after the drop, not  a little while before it.

Block Compression, DXTC, And Imaging

Imaging supported DXT image/texture compression since one of the earliest releases. Quality of compressed images isn’t very high though (at least the compression isn’t too slow). For future Imaging versions I plan to ditch the current compression code and add a new one. To be precise, two new ones – fast and lower quality (still probably better than current Imaging’s compression), and slow and higher quality mode. Fast one will be based on Real-Time DXT Compression by Id Soft. I’m not decided on high quality one yet, but probably something like cluster fit algorithm from Squish library.

Mainly for testing purposes during implementation of these new methods, I want to create extension for Imaging that compares two images (original and one reconstructed from compressed original) and measures PSNR and some other quality metrics.

I’m also thinking of implementing DXT5 based format using YCoCg colorspace and PVRTC (texture compression currently used in iPhone).

Few links if you’re interested:

Here’s some quick comparison of DXT compressors (click the image to see full size). Value in brackets is MSE (mean square error) – lesser number means compressed image is more similar to original.

DXT compressors comparison