Porting glSOAR to Android and OpenGL ES, Part 2

Porting glSOAR to Android and OpenGL ES, Part 1 [the linked post is not published yet] was more about used libraries and Java. Now part 2 tells a story of transition from OpenGL to OpenGL ES for glSOAR terrain renderer.

glSOAR OpenGL ES Gotchas

Initially, I wanted to just use fixed pipeline because that’s what desktop glSOAR uses (remember, original SOAR is from 2001). So that meant using GLES 1.0/1.1 since GLES 2.0 removed all the fixed pipeline stuff (matrix settings, lighting, immediate mode, and so on). To quote the official GLES docs:

Note: Be careful not to mix OpenGL ES 1.x API calls with OpenGL ES 2.0 methods! The two APIs are not interchangeable and trying to use them together only results in frustration and sadness.

I have to admit I didn’t really check what features GLES actually has, somehow assuming that it would be on par with regular OpenGL (1.x core or some 2.0). Here’s the list of few problems I encountered during the conversion:

  1. There’s no automatic texture coordinate generation. Desktop glSOAR uses OpenGL to generate texture coordinates for terrain mesh (by the means of glTexGen) to save memory. Fortunately, simple workaround is possible by setting texture transformation matrix directly (details at fernlightning). Of course, when using GLES 2 you can just generate coordinates in the shader.
  2. No wireframe display in GLES! There’s no glPolygonMode so you only get filled triangles. Desktop glSOAR can display wireframe overlay over textured terrain to show off cLOD in action by drawing terrain in additional pass with polygon mode set to GL_LINE and using glPolygonOffset. Now in GLES, I could try rendering the terrain as GL_LINES instead of GL_TRIANGLES. That kind of works so far (getting wire quads instead triangles though) for simple terrain grid but it will probably break when cLOD is implemented.
  3. Then I hit the show stopper, at least for GLES 1.0/1.1. There’s no support for 32-bit indices (GL_UNSIGNED_INT enum for glDrawElements) in GLES core. And 16-bit indices are any good only for terrains with size of 129×129 and smaller (as with SOAR it just cannot be simply split to smaller chunks). Fortunately, there is a GLES extension that allows usage of 32-bit indices called GL_OES_element_index_uint. I’ve seen on GLBenchmark page that my phone and many others (at least those with Andreno and PowerVR GPUs) support it but my test program insisted otherwise. As it turned out, it’s only supported with GLES 2 context. So it was a goodbye to GLES 1 and fixed function pipeline…
  4. The move to GLES 2 was actually quite easy since glSOAR just needs to output textured triangles with no fancier stuff. GLES GLSL shaders are a little different than regular OpenGL shaders. For instance, there are no predefined variables like gl_Vertex, gl_TexCoord, gl_ModelViewProjectionMatrix, etc. and there is only gl_Position and gl_FragColor for setting the results. Vertex positions, texture coordinates, and so on are passed to shader as attributes and transformation matrices as uniforms. Fortunately, GLES-style shaders work in desktop OpenGL without problems.

Some additional GLES findings

Good listing of supported GLES extensions for different phone and tablet models can be found at GLBenchmark Results (select model and then look at GL config tab).

Texture Compression

Some form of texture compression is supported by nearly all (if not all) current mobile GPUs. On desktop, it’s easy now and has been for many years. We have S3TC/DXTC (supported by GPUs for ages), its variant ATI 3Dc (uses alpha channel coding scheme from DXT5, supported by all DirectX 10 GPUs and older too), and recent addition BC6/BC7 formats by DirectX 11 class GPUs.

Unfortunately, it is not so easy in GLES and mobile GPU world. The problem is that each vendor can support completely different compressed formats. Only certainty is that GLES 2.0 capable GPU supports ETC1 (Ericsson TC) compression (no alpha channel). As far as Android is concerned, ~90% devices have GLES 2 GPU (as of Oct 2012). Additionally, S3TC is supported by Nvidia Tegra, PVRTC by PowerVR, ATI-TC/ATC by Andreno.

New ETC2 compression looks interesting though. It is part of the core of new OpenGL 4.3 as well as GLES 3.0. On desktop, it should be available for all DirectX 11 class GPUs (when the drivers arrive). The quality is supposedly better than S3TC and it has none of its patenting issues.

Anyway, for new glSOAR it looks like ETC1 for Android target and S3TC for desktop. Most probably in KTX (Khronos Texture).So that means writing KTX loader in Java and probably some ETC1 and KTX stuff for Vampyre Imaging Library too.

Some tools: etcpack tool from Ericsson for ETC1/ETC2 compression (outputs KTX files), etc1tool for ETC1 is part of Android SDK, and ATI Compressonator can compress ETC1, S3TC/DXTC, 3Dc, and ATI-TC.

NPOT Textures

Non-power-of-two textures have been supported by desktop GPUs for quite some time (at least all DirectX 10 capable GPUs have full support – not sure how “full” it is for example on Intel iGPUs). GLES 2 has limited support for NPOT textures (no mipmaps, limited texture wrapping modes, etc.) and with GL_OES_texture_npot extension you get full support for NPOT textures in GLES.

Current Status and Near Future

Now I have just a grid terrain rendering with no LOD done. Some parts of the SOAR code are translated to Java already. I’m quite worried about the SOAR LOD performace on Android though. Firstly, index buffer is rebuilt each frame (and can be quite big) and secondly, mesh refinement uses a lot of floating point calculations. Also the memory limit for Android apps is quite low for larger terrains (20 bytes are needed per vertex for SOAR plus a lot of 4-byte indices).

Well, if it is unusably slow, I can always abandon SOAR and do a Geomipmapping demo instead (that’s the plan for later anyway :)).

glSOAR Android Test running on Nexus S

Porting glSOAR to Android and OpenGL ES, Part 1

A few weeks ago I had some sort of a urge to try game and/or graphics development on Android. I’ve been doing some Android application development lately so I had all the needed tools setup already. However, what I wanted was some cross-platform engine or framework so that I could develop mostly on desktop. Working with emulators and physical devices is tolerable when doing regular app development and of course you want to use UI framework native for the platform. For games or other hardware stressing apps I really wouldn’t like to work like that. Emulator is useless as it doesn’t support OpenGL ES 2.0 and is generally very slow. And it’s no pleasure with physical device either, grabbing it all the time off the table, apk upload and installation takes time, etc. So I started looking for some solutions.

The Search

Initially, I considered using something like Oxynege. It uses Object Pascal syntax, can compile to .NET bytecode and also Java bytecode (and use .NET and Java libs). However, I would still need to use some framework for graphics, input, etc. so I would be locked in .NET or Java world anyway. I’m somewhat inclined to pick up Oxygene in the future for something (there’s also native OSX and iOS target in the works) but not right now. Also, it’s certainly not cheap for just a few test programs. Then there is Xamarin – Mono for Android and iOS (MonoGame should work here?) but even more expensive. When you look Android game engines and frameworks specifically you end up with things like AndEngine, Cocos2D-X, and libGDX.

After a little bit of research, I have settled for libGDX. It is a Java framework with some native parts (for tasks where JVM performance maybe inadequate) currently targeting desktop (Win/OSX/Linux), Android, and HTML5 (Javascript + WebGL in fact). Graphics are based on OpenGL (desktop) and OpenGL ES (Android and web). Great thing is that you can do most of the development on desktop Java version with faster debugging and hot swap code. One can also use OpenGL commands directly which is a must (at least for me).

The Test

I wanted to do some test program first and I decided to convert glSOAR terrain renderer to Java and OpenGL ES (GLES). The original is written in Object Pascal using desktop OpenGL. The plan was to convert it to Java and OpenGL ES with as little modifications as possible. At least for 3D graphics, libGDX is relatively thin layer on top of GLES. You have some support classes like texture, camera, vertex buffer, etc. but you still need to know what’s a projection matrix, bind resources before rendering, etc. Following is the listing of a few ideas, tasks, problems, etc. during the conversion (focusing on Java and libGDX).

Project Setup

Using Eclipse, I created the projects structure for libGDX based projects. It is quite neat actually, you have a main Java project with all the shared platform independent code and then one project for each platform (desktop, Android, …). Each platform project has just a few lines of starter code that instantiates your shared main project for the respective platform (of course you can do more here).

Two problems here though. Firstly, changes in main project won’t trigger rebuild of the Android project so you have to trigger it manually (like by adding/deleting empty line in Android project code) before running on Android. This is actually a bug in ADT Eclipse plugin version 20 so hopefully it will be fixed for v21 (you star this issue).

Second issue is asset management but that is easy to fix. I want to use the same textures and other assets in version for each platform so they should be located in some directory shared between all projects (like inside main project, or few dir levels up). The thing is that for Android all assets are required to be located in assets subdirectory of Android project. Recommended solution for libGDX here is to store the assets in Android project and create links (link folder or link source) in Eclipse for desktop project pointing to Android assets. I didn’t really like that. I want my stuff to be where I want it to be so I created some file system links instead. I used mklink command in Windows to create junctions (as Total Commander wouldn’t follow symlinks properly):

d:\Projects\Tests\glSoarJava> mklink /J glSoarAndroid\assets\data Data
Junction created for glSoarAndroid\assets\data <> Data
d:\Projects\Tests\glSoarJava> mklink /J glSoarDesktop\data Data
Junction created for glSoarDesktop\data <> Data

Now I have shared Data folder at the same level as project folders. In future though, I guess there will be some platform specific assets needed (like different texture compressions, sigh).

Java

Too bad Java does not have any user defined value types (like struct in C#). I did split TVertex type into three float arrays. One for 3D position (at least value type vectors would be nice, vertices[idx * 3 + 1] instead of vertices[idx].y now, which is more readable?) and rest for SOAR LOD related parameters. Making TVertex a class and spawning millions of instances didn’t seem like a good idea even before I began. It would be impossible to pass positions to OpenGL anyway.

Things like data for VBOs, vertex arrays, etc. are passed to OpenGL in Java using descendants of Buffer (direct) class. Even stuff like generating IDs with glGen[Textures|Buffers|…] and well basically everything that takes pointer-to-memory parameters. Of course, it makes sense in environment where you cannot just touch the memory as you want. Still it is kind of annoying for someone not used to that. At least libGDX comes with some buffer utils, including fast copying trough native JNI code.

Performance

Roots of SOAR terrain rendering method are quite old today and come from the times when it was okay to spend CPU time to limit the number of triangles sent to GPUs as much as possible. That has been a complete opposite of the situation on PCs for a long time (if did not have old integrated GPU that is). I guess today that is true for hardware in mobile devices as well (at least the new ones). And there will also be some Java/Dalvik overhead…

Anyway, this is just an exercise so it the end result may very well be useless and the experience gained during development is what counts.

OpenGL >> GLES

Continue reading to Porting glSOAR to Android and OpenGL ES [the linked post is not published yet] which focuses on OpenGL to OpenGL ES transition.

Deskew Tool Updated

There is a new version of Deskew command line tool introduced in post Deskewing Scanned Documents. Looks like quite a few people found it useful 🙂

What’s new in the latest version?

  • Background color can be defined (empty space around the original page after the rotation is filled with this color)
  • “Area of interest” rectangle to force skew detection only into selected part of the page (useful when  e.g. noisy page borders or images confuse skew detection when processing the entire page)
  • 64 bit and Mac OSX support
  • PSD and TIFF file format support (TIFF only in Win32 for now, sorry)
  • Display of skew detection stats and program parameters

Download

  Deskew 1.20
» 4.1 MiB - 7,206 hits - January 5, 2011 (last update November 1, 2016)
Command line tool for deskewing scanned documents. Binaries for several platforms, test images, and Object Pascal source code included.

Source Code Repository

Public Mercurial source repository of Deskew is now hosted at BitBucket: https://bitbucket.org/galfar/app-deskew.

Photo Sorting Tool

Let’s say you have just spent few weeks in some exotic country with a bunch of friends. Everyone had a digital camera and made thousands of photos. Now you have several directories with hordes of oddly named photos and you just want see all of them in the order they were taken. Throwing them all in one folder and setting file ordering by date might help but everyone’s camera can have a different internal time. File system dates can also be lost on their way to you (FTP upload etc.).

Being a programmer, rather that searching for some program on the Internet, I wrote my own quick and dirty command line tool for this task (mostly hard-coded paths etc.) in mid 2010. This year, I added GUI (where all the settings can be adjusted), some more date & time functions, and basically made the whole thing usable. And finally, beta release of PhotoMixer is available.
Continue reading

Enabling Repository Snapshots in Mercurial Browser at SF.net

Imaging’s code repository at SourceForge has been migrated from Subversion version control system to Mercurial. However, repository browser for Mercurial at SF.net doesn’t provide “download tarball” feature out of the box (that browser for SVN does). It’s a simple way for users that don’t want to install VCS clients they don’t normally use to download the current revision of your project in single archive.
Fortunatelly, enabling snapshot downloads for Mercurial is possible using SF.net’s shell service:

  • Login to your shell service at SF.net (ssh to username,projectname@shell.sourceforge.net) and navigate to your project’s repository. You start in your home dir (for me that’s /home/users/g/ga/galfar) and your project’s repo dir might look like /home/scm_hg/i/im/imaginglib/imaginglib/.hg/.
  • There you create a new file named hgrc (repository configuration) or edit the existing one.
  • Append the following two lines to that config file:
    [web]
    allow_archive = gz zip
  • Save the config and enjoy downloading zip and tar.gz archives! You’ll see new zip and gz items in the top menu of repo browser (e.g. summary | shortlog | changelog | graph | tags | branches | files | zip | gz) when viewing the revisions and also corresponding two icons in repository list that can be used to get the snapshot of top revision of each repository.

Hg repo browser witch snapshots

PS: You can convert whole repository to Mercurial with all the history and tags simply by using Convert extension. It takes quite a lot of time when you convert remote repository directly but the conversion continues where it left off when you get timeout or some network problem and run the conversion again.

16bit half float in Pascal/Delphi

Floating point numbers with 16 bits of precision are used mostly in computer graphics. They are also called half precision floating point numbers (as having half the bits of single precision 32bit floats). There’s one sign bit, five bit exponent, and ten bits for mantissa. Half floats are not really meant to be used for arithmetic computations due to the limited precision (and no support in common CPUs/FPUs).

Half floats first appeared in early 2000s as samples in images and textures. Floats provide higher dynamic range than what is available with regular 8bit or 16bit integer samples. On the other hand, commonly used single and double precision floats have much higher memory cost per pixel. Half floats have more reasonable memory requirements and their precision is adequate for many usages in imaging.

16bit float formats have been supported by ATI and NVidia GPUs for many years. I’m not sure about other IHVs but at least Direct3D 10 capable GPUs should all support it.

Read on if your interested how to convert between half and single precision floats (Object Pascal code).

Continue reading

Deskewing Scanned Documents

Check out updates and new versions of Deskew tool.

Some time ago I wrote a simple command line tool for deskewing scanned documents called Deskew. Technically, it’s a rotation since angles are preserved and skew transformation doesn’t do that. However, deskewing is commonly used term in this context.

Deskewing some smart paper

My approach is fairly common for this problem – rotation angle is first determined using Hough transform and then the image is rotated accordingly. Classical Hough transform is able identify lines in the image and it was later extended to allow detection of any arbitrary shapes.

Lines of text can be thought of as horizontal lines in the image. In a skewed scanned document all the lines will be rotated by some small angle. We can start with the equation of the line y = k · x + q. Since we’re interested in the angle, we can rewrite it as y = (sin(α) / cos(α)) · x + q. Finally, we can rearrange it as y · cos(α) − x · sin(α) = d. Now every point [x, y] in the image can have infinite number of lines going through it, where each is defined by two parameters: angle α and distance from the origin d.

We want to consider lines only for certain points of input image. Ideally, that would be the base lines on which the “text is sitting”. Simple way of determining these points is to check for black pixels which have white pixels just below them. Now for each of the classified points, we determine parameters α and d for all the lines that go through them. To get some finite number of lines, we calculate d for angles α from a certain range (I use angle step of 0.1 degrees). We want to find a line that intersects as many classified points as possible – an accumulator is used to store “votes” for each calculated line. For each point that is believed to be on the text base line, we add one vote for each line that intersects it. At the end, we find the top lines that have the most votes. Ideally, these are the base lines of all lines of text in the document. Finally, we get the rotation angle by averaging angle α of the top lines and rotate the whole image accordingly.

Important part is that one: “check for black pixels which have white pixels just below”. What’s black and white is determined by comparing value of the current pixel against some given threshold. For images where background is plain white and the text is black it’s easy just to use 0.5 as the threshold. But when the background/foreground distinction is not so sharp calculating the threshold adaptively based on the current image can be very useful. Deskew supports both adaptive threshold calculation as well as specifying constant threshold as command line parameter.

Deskewing some math exercise

Implementation is written in Object Pascal and uses Imaging library for reading and writing various image file formats. There are precompiled binaries for a few platforms, others be built from sources using Free Pascal compiler. Archive also contains few test images.

  Deskew 1.20
» 4.1 MiB - 7,206 hits - January 5, 2011 (last update November 1, 2016)
Command line tool for deskewing scanned documents. Binaries for several platforms, test images, and Object Pascal source code included.

Lego Space

I was a big fan of early 90s Lego Space sets when I was a boy. It was a time of M-Trons, Blacktrons, Space Police, Ice Planet, and others. Unfortunately, Lego was very expensive compared to local wages back then so I didn’t get many of them, let alone some big ones.

Much later, in 2008, I stumbled upon small Lego exhibition made by some local fans. There was a really huge M-Tron base and it got me thinking about buying few of those larger Space sets I wanted years ago. Well, after about two-month eBay shopping spree I had all M-Tron sets – together with about 60 others (and some individual parts orders from Bricklink). I became a nightmare for local post women with all these international packages coming nearly every other day. My best buy was probably Blacktron Message Intercept Base with original box for less than £20 from eBay UK.

These days I don’t have time for Lego anymore, but hopefully this will change some day. And some larger building space would be nice too!

My latest creation is Ice Planet’s Deep Freeze Defender rebuilt as a M-Tron ship. Unfortunately, it was quite damaged during moving to a different house so I’ll have to do some repairs.

Check out my Brickshelf gallery and current set collection.