Debugging with CDB in QtCreator Agonizingly Slow

A few years ago I wrote about my Problems with CDB Debugger in QtCreator. And a few months ago debugging started getting slower and slower and it finally reached a point where the program would need a minute or more just to start and stopping for a breakpoint was almost a coffee break situation. The old remedy wasn't helping anymore — this was something new.

While searching for a cure I stumbled upon Qt mailing list thread where the culprit was identified. It's Microsoft's symbol server!

Just go to Options -> Debugger -> CDB Paths tab and make sure srv* path is below cache* (or just remove it altogether). Looks like server symbols are preferred over local symbol cache if on top and that server is now terribly slow.

Deskew Tool v1.25 Released

New version of Deskew command line tool is ready. You can find general info about Deskew here Deskew Tools.

Change List for Deskew 1.25

  • fixed issue #6: Preserve DPI measurement system (TIFF)
  • fixed issue #4: Output image not saved in requested format (when deskewing is skipped)
  • dynamic loading of libtiff library - adds TIFF support in macOS when libtiff is installed
  • fixed issue #8: Cannot compile in Free Pascal 3.0+ (Windows) - Fails to link precompiled LibTiff library
  • fixed issue #7: Windows FPC build fails with Access violation exception when loading certain TIFFs (especially those saved by Windows Photo Viewer etc.)
  • Linux ARM build is now also included in the release

Download

  Deskew v1.30
» 4.3 MiB - 20,632 hits - June 19, 2019
Command line tool for deskewing scanned documents. Binaries for several platforms, test images, and Object Pascal source code included.

Deskew Tool v1.20 Released

New version of Deskew command line tool is ready. You can find general info about Deskew here Deskew Tools.

Change List for Deskew 1.20

  • much faster rotation, especially when background color is set (>2x faster, 2x less memory)
  • can skip deskewing step if detected skew angle is lower than parameter (possible speedup when processing large batches)
  • new option for timing of individual steps
  • fix: crash when last row of page is classified as text
  • misc: default back color is now opaque black, new forced output format "rgb24",
    background color can define also alpha channel, nicer formatting of text output

Download

  Deskew v1.30
» 4.3 MiB - 20,632 hits - June 19, 2019
Command line tool for deskewing scanned documents. Binaries for several platforms, test images, and Object Pascal source code included.

GEOTRANS in .NET and Java

GEOTRANS

GeoTrans (Geographic Translator) is a software that allows you to convert geographic coordinates among a variety of coordinate systems, map projections, and datums. It is made available by U.S. National Geospatial Intelligence Agency at no cost and free of copyright restrictions.

It is written in C (recent versions are in C++) and I've seen it used in software inside various surveying instruments. I worked on a supporting software for these devices and needed to do conversions of measured coordinates etc. So naturally, I wanted to use the tried and tested GeoTrans in my software as well. Only I needed to use it in .NET and Java…

Continue reading

Problems with CDB Debugger in QtCreator

Some time ago Locals and Expressions view in QtCreator just stopped working for me. No locals were listed when the program stopped on a breakpoint, watches did not show the values, hovering with the mouse over a variable in the editor did not show its value in a tooltip. No fiddling with IDE options helped. This was on Windows using compiler from Visual C++ and CDB as a debugger.

It was quite annoying but I was mostly working on UI at that time and could live with occasional dumping of few variables into a log. Few weeks later I moved to some math heavy stuff and the situation became desperate — I had to fix this!

At first, I suspected the debugger or the compiler. Since reinstalling and trying different versions did not help I also tried different versions of Qt and QtCreator itself. Still broken. And Googling for broken CDB only revealed problems with CDB getting unusably slow.

After QtCreator reinstall, I noticed the settings were preserved from the old version. So maybe they are somehow corrupted?  I found them in C:\Users\username\AppData\Roaming\QtProject\qtcreator\, made a backup of that folder, and deleted it. And it started working! Locals and Expressions were back!

Of course, I also wanted my old settings back. So I switched back to the old settings and started QtCreator again and to my surprise it still worked! I took a closer look and saw that the problem lies in the session file (*.qws) — when I previously deleted the settings folder QtCreator created a new session for me and this one I reopened in the IDE after the switch back to the old settings (just copied the files into the folder so the new session file stayed there).

So I opened the session file with an intention to start deleting suspicious stuff until it works. It's a XML file and apart from one big base64 blob there is not much else. Besides a breakpoint list, the only thing that seemed related to debugging was a list of maybe ten old watch expressions (which were not listed in the IDE anymore!). I deleted it and voilà, I could see the values of locals and variables again!

After a few weeks, debugging started to get slow. Stepping over code took a few seconds and values of the variables showed up only after considerable time (tens of seconds). This time I knew what to do, deleting watch expressions from the session file helped again. So I came to the conclusion that in the first case the values could eventually show up after a few minutes (stepping over code was not slowed down though). It was just very very slow. Then the Googled complaints about slow CDB made more sense to me. And indeed, I found out someone fixed it the same way but I was not paying attention to just slow debugging before, I was looking for broken debugging!

Recapitulation

  1. Debugging in QtCreator using CDB in Windows is very slow and/or values of locals and expressions never show up.
  2. Go to the folder where QtCreator stores your session file (C:\Users\username\AppData\Roaming\QtProject\qtcreator\).
  3. Open the session file (*.qws) in text editor and look for this XML subtree:
    <data>
        <variable>value-Watchers</variable>
        <valuelist type="QVariantList">
            <value type="QString">some expression 1</value>
            <value type="QString">some expression 2</value>
            <value type="QString">some expression 3</value>
        </valuelist>
    </data>
    
  4. Delete this subtree (or just the expressions) and save the file.
  5. Start QtCreator, open the fixed session, and do some debugging.

Edit: If this doesn't help check also Debugging with CDB in QtCreator Agonizingly Slow.

.NET and Java: Generating Interoperable AES Key and IV

Let's assume we want to generate encryption key and initialization vector (IV) for AES encryption based on some passphrase. And we want to be able to generate the same key and IV for the same passphrase in .NET and Java - maybe we have Android app written in Java that needs to decrypt message from ASP.NET web app.

In .NET, Rfc2898DeriveBytes class is often used to derive keys of specified length according to given passphrase, salt, and iteration count (RFC2898 / PBKDF2). For 256-bit key and 128-bit key it is as simple as this:

var keyGen = new Rfc2898DeriveBytes(passwordBytes, 
    saltBytes, iterationCount);

byte[] key = keyGen.GetBytes(256 / 8);
byte[] iv = keyGen.GetBytes(128 / 8);

Fortunately, PBKDF2 implementation is also built-in in Java:

SecretKeyFactory factory = 
    SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
KeySpec spec = new PBEKeySpec(passwordChars, 
    saltBytes, iterationCount, 256);
SecretKey secretKey = factory.generateSecret(spec);
byte[] key = secretKey.getEncoded();

We have the same key byte array, albeit with some more typing. And how about the initialization vector now? One could think that creating new PBEKeySpec with a length of 128 is the way to go. I know I did.

However, you would just get the same bytes as for the key (the first half of them). This key derivation algorithm is deterministic so for the same inputs you get the same output. Each call of GetBytes of .NET's Rfc2898DeriveBytes just returns more and more bytes generated by the algorithm whereas Java implementation needs to know the total output length upfront. So for 256-bit key and 128-bit IV we need to create PBEKeySpec with the length of 384 and split the result between key and IV:

KeySpec spec = new PBEKeySpec(passwordChars, 
    saltBytes, iterationCount, 256 + 128);
SecretKey secretKey = factory.generateSecret(spec);

byte[] data = secretKey.getEncoded();
byte[] keyBytes = new byte[256 / 8];
byte[] ivBytes = new byte[128 / 8];

System.arraycopy(data, 0, keyBytes, 0, 256 / 8);
System.arraycopy(data, 256 / 8, ivBytes, 0, 128 / 8);		

Note: All the Java stuff was tested only with Android.

Deskew Tool Version 1.10

New version of Deskew command line tool is ready. You can find general info about Deskew here Deskew Tools.

Change List for Deskew 1.10

  • TIFF support now also for Win64 and 32/64bit Linux platforms
  • forced output formats
  • fix: output file names were always lowercase
  • fix: preserves resolution metadata (e.g. 300dpi) of input when writing output

Continue reading

Android Terrain Rendering: Vertex Texture Fetch, Part 1

To my surprise, I found out that GPU (PowerVR SGX 540) in my venerable Nexus S (2010) supports vertex texture fetch (VTF). That is, accessing texture pixels in vertex shader -- a very useful feature for terrain rendering. About a year ago, when I started investigating terrain rendering on Android devices, I did some searching for VTF support on Android devices and figured out it's not there yet (similar to situation years ago when desktop OpenGL 2.0 was released with support for texture sampling in GLSL verter shaders but most GL implementation just reported GL_MAX_VERTEX_TEXTURE_IMAGE_UNITS to be zero). Now I don't know how I missed it on my own phone, maybe there was some Android update with updated GPU drivers during last year? I have no idea how many other devices support it now. Hopefully, newest ones with OpenGL ES 3 support it all. I wouldn't be surprised if among GLES 2 devices only PowerVR + Android 4+ combinations supported it.

Overview

Anyway, let's focus on terrain rendering - rough outline:

  1. Put the entire heightmap into a texture.
  2. Have a small 2D grid (say 16x16 or 32x32) geometry ready for rendering terrain tiles.
  3. Build a quad tree over over the terrain. Root node covers entire terrain and each child then covers 1/4th of parent's area.
  4. Now we can start rendering, do this every frame:
    1. Traverse the quadtree starting from the root.
    2. For each child node test if the geometry grid provides sufficient detail for rendering the area covered by this node:
      • YES it does, mark this node for rendering and end traversal of this subtree.
      • NO it does not, continue traversal and test children of this node (unless we're at leaf already).
    3. Now take this list of marked nodes and render them. The same 2D grid is used to render each tile: it's scaled according to tile's covered area and its vertices are displaced by height values read from texture.
Root covers entire terrain, each child quarter of the parent's area.

Root covers entire terrain, each child quarter of the parent's area.

This is basically what I originally wanted for Multilevel Geomipmapping years ago but couldn't do in the end because of the state of VTF support on desktop at that time.

So what exactly is the benefit of VTF over let's say geomipmapping here?

Main benefit is ability to get height of the terrain at any position (and multiple times) when processing each tile vertex. In traditional geomipmapping, even if you can move tile vertices around it's no use since you have only one fixed height value available. With VTF, you can modify tile position of vertex as you like and still be able to get correct height value. This greatly simplifies tasks like connecting neighboring tiles with different levels of detail. No ugly skirts or special stitching stripes of geometry are needed as you can simply move edge points around in the vertex shader. Also geomorphing solutions begin to be usable without much work. And you can display larger terrains as well. With geomipmapping you always have to draw fixed number of tiles (visible leaves) -- a number that goes up fast when you enlarge the terrain. VTF may allow you draw fixed number of tiles -- regardless of actual terrain size (as distant tiles cover much larger area compared to geomipmap tiles with fixed area). Another one, terrain normals can be calculated inside the shaders from neighboring height values.
Finally, since heightmap is now a regular texture you get filtering and access to compressed texture formats to stuff more of the terrain to the memory.

There must be some disadvantages, right?

Sure, the support for VTF on mobile GLES 2 GPUs is scarce. So for something else than tech demo it's useless for the time being. Hopefully, all GLES 3 GPUs will support VTF. And with usable performance - VTF was uselessly slow on desktops in the beginning.

Implementation

I have added experimental VTF based terrain renderer to Terrain Rendering Demo for Android testbed and it looks promising. Stitching of the tiles works flawlessly. More work is needed on selecting nodes for rendering (render node or split to children?). Currently, there's only simple distance based metric but I want to devise something that takes classical "screen-space error" into account. And maybe some fiddling with geomorphing on top ...

Follow some of the implementation details in part 2 (soon!).

VTF Terrain Shot

.NET and WPF Notes #1

I have been working in .NET and WPF lately. Of course, I ran into some issues and had to look up some solutions. I wrote some of it down, for "future reference" and for anyone who might be interested.

Design Time Data and Properties

When designing WPF data template for ListBox items I wanted some preview in design time with sample mock data for the items. Also, some simple way to override certain property values would be nice (e.g. when you bound brushes of path to some run time values but want to use fixed ones for design). To use the mock data you can use d:DataContext design time attribute. Some class with the sample data is created in code behind and then bound using this attribute.

public class MockMeasurementList
{
  private MeasurementList measurements = new MeasurementList();
  public MeasurementList Measurements { get { return measurements; } }

  public MockMeasurementList()
  {
    measurements.Add(new Measurement(
      new Position(49.3051356, 16.5607972, 358),
      new Vector(52.035075, 7.854967, 1492.690400));
  }
}
<UserControl.Resources>
    <mocks:MockMeasurementList x:Key="DesignList"/>
</UserControl.Resources>
<ListBox Name="ListMeasurements"
   d:DataContext="{Binding Source={StaticResource DesignList}}"
   ItemsSource="{Binding Measurements}">

WPF design time value for simple properties like brushes and sizes can be done for example using this approach of Marcin Najder. I use it for example for stroke and fill brushes of map markers:

<UserControl ... xmlns:dtools="clr-namespace:DesignTools">
  ...
  <Path Stroke="{Binding Stroke}" Fill="{Binding Fill}"
        dtools:d.Stroke="Navy" dtools:d.Fill="PowderBlue">
  ...

Events and Delegates Returning Bool

Suppose we have an event like this:

public event Func<bool> StoreMeasurementQuery;

and we want to do some action (e.g. store a measurement) only if all handlers subscribed to the event return true (Is the measurement valid? Is there enough storage?). Now if we raise the event the usual way (assuming StoreMeasurementQuery != null):

bool store = StoreMeasurementQuery();

the result won't be as one could expect as store will hold the return value of only the last handler executed. To get some sensible results we have to execute the handlers individually and check out their return values. Based on this SO answer I wrote two these extensions:

public static bool AllSubscribersReturnTrue(this Func<bool> evt)
{
  return evt.GetInvocationList().Cast<Func<bool>>().
    Select(func => func()).ToList().All(ret => ret);
}

public static bool AnySubscriberReturnsTrue(this Func<bool> evt)
{
  return evt.GetInvocationList().Cast<Func<bool>>().
    Select(func => func()).ToList().Any(ret => ret);
}

to get logical AND and OR of all event handlers return values.

Grid Mouse Interaction

By default, Grid and other Panel controls don't receive mouse events. If you want to use Grid for example as clickable container for list box items, you need to set some background to it. Background="Transparent" is good enough.

Google Apps Script Notes

Extend, Inherit

I did some work in Google Apps Script for a friend recently. After a while, I had a number of disorganized helper functions that were basically extensions for various objects from GAS Default Services (mostly batching more methods together, etc.). Later, I started wondering if there is a way to extend GAS objects directly. Unfortunately, there is currently no way to do that. And according to this issue with won't fix resolution there will never be one.

So I decided to put the related functions to separated classes. And I wanted some sort of inheritance so that I could have something like this:

// Create GAS object
var table = uiApp.createFlexTable();
// Some basic table writer for generic usage
var writer = new TableWriter(table);
writer.writeRow(["a", "b", "c"], style);
// Specialized table generator, subclass of generic TableWriter
var builder = new ReportBuilder(table);
builder.addOrder(order);

Continue reading