Fixed a two bugs in the digup tool: added large file support when compiling the program and fixed a string allocation bug.

This new version enables large file support by using long long variables for size. Furthermore, a string allocation bug was fixed which occured when using -t and -f command line parameters.

For more information and the new version see the digup web page.

Fixed a small bug in the stx-execpipe library: add large file support when compiling the library.

This bug switches on the large file support functions. Without this fix a pipeline reading or writing files >2GB will not function properly. The fix is to #define _FILE_OFFSET_BITS 64 when compiling the library's code.

For more information and the source code see the stx-execpipe web page.

The STX C++ library series has been extended today with a new installation: the STX Execution Pipe library, in short STX ExecPipe. It is the solution to an issue that I encountered in writing a small backup application. This backup tool collects some file to backup and then calls tar and xz followed by ncftpput.

However, I could not find any useful C++ library that allows convenient chaining of external programs like used in everyday shell piping. This pipe line functionality is based on the system calls fork(), exec() and pipe(), which are not easy to use correctly. After writing some ad-hoc functions to call one or two external programs, I decided to tackle this basic problem once and for all. The result is the stx-execpipe library.

Using the library a C++ program can build a sequence of external programs with command line parameters. These programs are connected by the library just like a shell pipe: stdout of the preceding stage goes into stdin of the next one. The input and output of the whole pipeline can be redirected to a plain fd, a file or saved in a std::string.

One very interesting feature of the library is to insert intermediate processing functions into a pipe line. The data can be intercepted and passed back to the parent process for manipulation or just inspection. This was necessary to calculate the SHA1 digest of a backup tarball simultaneously to uploading it.

For more information and the source code see the stx-execpipe web page.

The following small code snippet exemplifies the flexibility of the stx-execpipe solution:

Published yet another C++ template library using a B-tree. This time the solution is a disk-based read-only key-value mapping using a "packed, sequential" B-tree as index structure.

All applications mapping a large number of constant, integral keys to string or data blobs can benefit from this library. The database structure is highly compact and contains self-verification checksums over both key and value areas.

stx-cbtreedb is a direct contender with cdb and tinycdb, which however are based on hash tables and do not retain key proximity. Compared to other full-fledged B-tree implementations like BerkeleyDB or TokyoCabinet, the stx-cbtreedb is very small, faster and the database files have much less overhead due to read-only optimizations.

For more information and the source code see the stx-cbtreedb web page.

My second LibriVox recording is finished! The basic idea behind LibriVox is to read public domain texts and to put the recordings back into the public domain. More about that on librivox.org.

As with the last recording, my personal motivation is to practice my rusty English pronunciation. By reading and rereading the texts I believe my English will become more fluent and in the end also achieve better articulation.

The second recording is "Double Take" by Richard Wilson, a science fiction short story available from Project Gutenberg. I have also made a quick LaTeX typesetted version of the Gutenberg etext for more comfortable reading and with slight corrections: doubletake_wilson_text.pdf (369 KB).

Story Summary / Teaser

Pacing through a high-speed spy adventure, young Paul Asher finds himself going around in circles -- very peculiar circles indeed!

Audio Recording - Runtime: 26:29

MP3 encoded with standard lame VBR preset (obviously better/smaller than CBR 128 kbps)

doubletake_wilson_tb_vbr.mp3 (19.1 MB)

The recording is now also available on archive.org in the LibriVox Short Science Fiction Collection 033.

Published a small, but very useful console tool to update md5sum or shasum digest files. It will read existing md5sum.txt files and add new files to it without rereading the whole directory tree.

This makes digup very useful to update and verify incremental archives like chronological data storages or music collections, which are nowadays commonly stored and backuped on hard disks. Using a full file digest scan even slowly creeping bad blocks on old hard disks can be detected. By using a crontab entry, this check can be performed unattended and routinely.

For more information, the source code and binaries for various platforms see the digup web page.

After two weeks of hard work, my first LibriVox recording is finally finished! The basic idea behind LibriVox is to read public domain texts and to put the recordings back into the public domain. More about that on librivox.org.

My personal motivation behind recording, besides a lasting inclination towards fantasy and science fictions texts, is to practice my rusty English pronunciation. By reading and rereading the texts I believe my English will get more fluent and in the end also achieve better articulation.

The first recording is "The Last Evolution" by John W. Campbell, Jr. a science fiction short story available from Project Gutenberg. I have also made a quick LaTeX typesetted version of the Gutenberg etext for more comfortable reading and with slight corrections: lastevolution_campbell_text.pdf (153 KB).

Story Summary / Teaser

A future Earth population is unexpectedly attacked by powerful extraterrestrial Outsiders. However, the highly developed civilization of humans and machines quickly builds up forces and defends itself. This desperate, life threatening struggle against the invading aliens leads to The Last Evolution.

Audio Recording - Runtime: 58:42

MP3 encoded with standard lame VBR preset (obviously better than CBR)

lastevolution_campbell_tb_vbr.mp3 (42.7 MB)

If someone drops me a line with a good reason for uploading 250 MB, then I can also provide a lossless 24-bit FLAC file. Just burning a CD is not a good enough reason.

The recording is now also available on archive.org in the LibriVox Short Science Fiction Collection 026.

Released a minor updated source code version for Flex Bison C++ Example. The example source code is released into the public domain or, at your option, under the Do What The Fuck You Want To Public License (WTFPL).

This minor bugfix release fixes up two simple compilation issues with the newest bison version 2.4.1.

For more information and the download package see the Flex Bison C++ Example web page.

While publishing screenshots for CryptoTE (nice cross-platform text editor with integrated cryptography) the idea of making a short tutorial screencast came up. A screencast is basically a video made directly from the computer screen with no camera involved. Usually speech comments are added to create a sort of impromptu screen presentation. Thus I could show off the cool features of CryptoTE, and people could see what the program is about without installing it.

So far the cool and very trendy idea, now for sorting out the technical difficulties in making such a video. In theory a screencast video should be well compressible with a lossless codec. Unresized screenshots in full-resolution are also better compressed using the lossless PNG format than with JPEG, the PNG files are much smaller and show no blurry edges. Similarly screencasts, being only a sequence of full-resolution, losslessly recorded screenshots with some little mouse-movement, should be compressed losslessly.

On the digital inspiration blog I found a detailed "Screencasting Software Guide - Review of Desktop Screen Recording Tools" for all platforms. However, because my primary operating system is Linux (and since I also wanted to show that in the screencast), the range of tools that actually work is very limited.

Capture Tools: Tried xvidcap, vnc2swf. Picked vncrec!

The first tool that popped up while searching was xvidcap. It is not related to the XviD codec, but derives its name from X Window. It is a small tool that can capture any activity on the X Window desktop into a variety of different formats. Among these formats are MPEG, MPEG2 and many other compressed formats supported through ffmpeg.

However, after a few trials it became clear that xvidcap was too slow. Even though it captured the screen only with 10 fps (frames per second), the encoding process or capture process itself was not fast enough for a perfect recording of screen activity. I had large frame drop rates and moving the mouse cursor didn't look very good with only 10 fps. I presumed that with so much computer power in present-day CPUs, a higher quality had to be achievable somehow. Some other minor problems occurred with xvidcap: dialogs popping up in the wrong places, and context menus went outside of the capture area.

After some research I found a very good solution: VNC (Virtual Network Computing) recording. Using VNC one can basically connect to a remote desktop and control the mouse and keyboard via the network. Due to prior experience with TightVNC, it was my software of choice and I quickly set up a secondary user account on my desktop computer. Another advantage of VNC and the secondary user account was that I could easily configure the X desktop to look pretty for recording at the uncommon resolution 800x700 pixel, which was needed to show all dialogs without clipping.

For recording VNC sessions there are multiple solutions, of which vnc2swf was the most commonly mentioned for creating screencasts. However, I had only lots problems with this software; the output files were enormous, there was no easy support for adding simultaneous audio and the basic version didn't even have a seek bar to jump back and forth in the video. Nevertheless, the idea of encoding RFB (Remote Framebuffer, the protocol behind VNC) messages as Flash operations is very good.

Much better suited for my purpose was vncrec. It fires up a normal VNC client session and records all mouse, keyboard and X11 display events in a .vnc file. Thus it essentially only records changes to the displayed desktop, exactly what I needed, and even in lossless quality. The .vnc files can be replayed using the -play directive.

Encoding: Examined Lossless Codecs, Resorted to MEncoder and x264

Next step was to convert the .vnc recordings into video files that are playable using a Flash video player. My favorite Flash video player is Flowplayer and a comprehensive guide to installing it is available on their website. Since Flowplayer uses Flash to play the video, the available codecs are limited to those supported by Flash. I discovered that Flash has a special lossless codec for "screen videos", it is labeled flashsv in the ffmpeg formats list. However, a few tests showed it to be way larger than a well compressed H.264 video. Using other encoding tools than ffmpeg did not show any improvement.

So I basically gave up on creating a lossless compressed version and resorted back to x264, because it is the currently most advanced free codec and H.264 video is supported by Flash. Thus the .vnc must be compressed to a .mp4 file. For this purpose I used the mencoder tool, part of the MPlayer suite. Mencoder cannot read .vnc files directly. Apparently transcode, an alternative Linux video tool suite, can read vncrec files via a plugin, however, the resulting quality was not agreeable. There seemed to be some problems in the RGB to YUV2 color-space conversion of transcode which resulted blurred dialog box edges.

The final solution was then to use an enhanced fork of the vncrec tool by twibright labs. This version contains some YUV4MPEG2 export code which produces a good looking YUV2 raw stream. These streams can be directly piped into mencoder, without any intermediate files.

Vncrec has no support for recording audio simultaneously, so that must be done with an extra audio tool like Audacity or plain ALSA arecord. Using mencoder the resulting audio file can be encoded using FAAC and then multiplexed alongside the H.264 video stream.

Final Sequence of Commands

First launching a VNC server on the secondary user account. More work is needed to get a complete desktop up and running. The following few lines are only the basic steps to start the VNC server display on :1. SSH is used instead of su because otherwise lots of environment variables are kept from the primary user.

$ ssh <recording-user>@localhost
Password: <enter password>
$ vncserver :1 -geometry 800x700 -depth 24 -deferupdate 0

You will require a password to access your desktops.

Password: <some temporary password>
Verify: <same again>
Would you like to enter a view-only password (y/n)? n

New 'X' desktop is box:1

Starting applications specified in ~/.vnc/xstartup
Log file is ~/.vnc/box:1.log

$

Returning to the primary user account, one can connect to the VNC server using the recording client, or any other VNC client for that matter:

$ vncrec :1
VNC server supports protocol version 3.8 (viewer 3.3)
Password: <temporary password from above>
VNC authentication succeeded

After setting everything up for recording, connect using the VNC capture program.

$ vncrec -record screencast.vnc :1
VNC server supports protocol version 3.8 (viewer 3.3)
Password: <temporary password from above>
VNC authentication succeeded

Now to do the recording. All VNC actions are saved to screencast.vnc. Audio is not automatically recorded as well. To capture speech, plug in a microphone and record it simultaneously using an audio recording software. I found it convenient to first start audio and then VNC, and later synchronize both by deleting sound prior to the ENTER key following the VNC password entry.

To terminate the recording, press F8 and select "Quit Viewer" or just close the VNC window. This will correctly stop recording and flush the capture file.

After recording, the VNC server can be terminated. As the secondary user:

$ vncserver -kill :1
Killing Xvnc process ID 24148

The resulting VNC recording can be replayed like this:

$ vncrec -play screencast.vnc

Prior to encoding, the VNC recording can also be viewed directly by piping the raw YUV2MPEG into MPlayer, note the switch telling MPlayer the stream type:

$ VNCREC_MOVIE_FRAMERATE=25 vncrec -movie screencast.vnc | mplayer -demuxer y4m -

For stream editing it is most useful to create a raw file without delta frames. For this purpose the HuffYUV or FFmpeg HuffYUV codecs are best used. HuffYUV files can also be processed by Windows software.

$ VNCREC_MOVIE_FRAMERATE=25 vncrec -movie screencast.vnc | mencoder -ovc lavc -lavcopts vcodec=ffvhuff -o screencast-ffvhuff.avi -demuxer y4m -

Finally the video and speech audio file can be encoded using x264 and FAAC into a .avi file.

$ VNCREC_MOVIE_FRAMERATE=25 vncrec -movie screencast.vnc | mencoder -ovc x264 -x264encopts crf=23 -oac faac -faacopts quality=100 -o screencast-temp.avi -demuxer y4m - -audiofile screencast-speech.wav

For display on the web, the .avi file must then only be repackaged as a .mp4 Matroska container for display on the web using a Flash player.

$ MP4Box -aviraw video -out screencast screencast-temp.avi
$ MP4Box -aviraw audio -out screencast screencast-temp.avi
$ MP4Box -new -add screencast_video.h264 -add screencast_audio.aac -fps 25 screencast.mp4

Summary

With vncrec great screencasts can be made on Linux using a VNC server. However, the available lossless video compression techniques are not as far developed as lossy movie compression, so I resorted back to using the x264 codec. A special patched version of vncrec allows easy piping of the uncompressed picture stream into MPlayer. Using mencoder this raw stream can be converted into many different formats and simultaneous speech can be also added.

In the end most work when into scripting and recording a fluent screencast video, because the post-production editing facilities available are not very rich. The only tools available for that are Avidemux and Audacity. Maybe you will have more luck with Cinelerra than I did.

You can see the finished tutorial screencast about CryptoTE by following the image link below:

After almost one year of personally testing the program, I decided to publicly released the first version of CryptoTE v0.5.390. One year in the making, CryptoTE is a very useful little text-editor with integrated cryptography. The name stands for CRYPTOgraphy Text Editor and it transparently encrypts text files storing them into secure containers. The program incorporates the popular editing component Scintilla and makes heavy use of wxWidgets.

For more information, the source code and binaries for various platforms see the CryptoTE web page.