Mick McCarthy

Useful RPM commands in Mandriva

2009-05-12T17:52:00.006+02:00

Mandriva provides some very nice commands which wrap rpm and allow you to install/remove packages directly from the repositories and query package information. Heres a short list of some the more useful commands....

urpmi blahblah
Install package blahblah.

urpme blahblah
Remove package blahblah.

urpme --auto-orphans
Remove all orphaned packages (I believe this is a recent addition).

urpmi --auto-update
Update all packages which have update available from one of the update repositories.

urpmi --clean
Clean the package cache on the local harddrive. Useful if a package gets corrupted during download and you cannot install it. Each successive attempt to repeat the install will use the cached version - gotta clean that cache first.

urpmi --test blahblah
Tests the install of a package without actually installing anything.

urpmq --summary blahblah
Display summary information about package blahblah.

urpmf regex
List all packages which contain a file whose name matches the regular expression regex. regex can therefore be the full filename, a partial filename or a more complex regular expression.

urpmf -f regex
Lists all packages as before and also displays the full package version and architecture information.

urpmq regex
Find if a package exists which has a name exactly matching regex. The parameter regex can be a Perl like regular expression.

urpmq -i regex
Display detailed information about a package.

urpmq -a regex
List all packages whose name fully or partially matches the regular expression regex. Useful when you don't know the full name of a package.

urpmq -f regex
Display the package name, version and architecture of matching package.

urpmq -l regex
List all files in matching package.

rpm -q blahblah
Check if package blahblah is installed.

The raw rpm command does allow you to query package information however the package must already be installed.

rpm -qa
List all installed packages.

rpm -ql blahblah
List the files in installed package blahblah

rpm -qi blahblah
Display detailed information about the installed package blahblah.

rpm -qR blahblah
List the packages which package blahblah depends on.

Payment Processing for the ISV

2009-05-07T16:03:00.004+02:00

I've recently completed a review of the services available for Independent Software Vendors who distribute their products over the internet. Here is a brief list of the various companies I came across during the review. I haven't gone into the details of pricing and features as these change over time, however this list should help you encounter the key service providers.

Digital River is a large US company which provides e-commerce solutions to software vendors. They offer sales, licensing and payment processing services to large vendors such as Symmantec and Kaspersky. They also provide these services to smaller vendors, shareware providers etc., marketing their platform through a range of websites. Their sites include RegNow, RegSoft, esellerate and Share-IT.

E-Junkie provide order processing, license key distribution, downloads but do not provide payment processing. Their platform integrates with PayPal, Google Checkout and other payment processors. For this reason, they charge a flat monthly fee and the payment processors percentage fee is charged independently. E-Junkie are based in Arizona.

FastSpring provides payment processing and distribution to software vendors. They are based in California.

Kagi also provide payment processing and software distribution services.Yes, they are also based in California.

There are several issues you should have in mind when choosing a company which provides these services:

Exclusivity. Are you free to sell your software product through other channels concurrently or do their terms and conditions restrict you to selling through their website?
Fraud Prevention. How good is their fraud prevention? You don't want a high proportion of your sales to consist of providing the license key or software download and then a couple of days later being informed that the payment was canceled because the purchaser's credit card was fraudulently used.
Chargebacks. If a user complains to the credit card company about not receiving the product or the product not being fit for purpose - how does the payment processor communicate with you and the customer?
Fees. How do their percentage fees compare to other providers? What is the minimum charge? Are you charged per download or for storage space?
Taxes. How does the service provider handle sales tax rates or EU VAT rates?
Funds Transfer. How do you get your funds from the service provider? Must you pay a fee for funds transfers? How frequently does the vendor receive their funds - monthly, bi-weekly, daily?

Bug Tracking for the Lone Developer

2009-01-30T10:15:00.008+01:00

Recently, I was looking for an bug and issue tracking application which I could install on my workstation. I need it to manage the issues associated with a fairly large and complex project but I'm not sharing development with anyone on this project so no support for multiple users or remote access is required. There are only a few open source issue tracking applications available which are mature enough to support a large project.
Some time ago I started using Trac. Trac is a web application, written in Python, which requires a web server and database server to be installed. It provides Ticket management and Subversion integration. Tickets can be organised into a pseudo-schedule using Milestones. Unfortunately, most of Trac's configuration is achieved on the command line - there is no administrator interface. This is OK for a very rudimentary project, however, it gets very tedious once the number of tickets starts to grow. (On a side note, I was able to find a Mylyn plug-in for Trac which allowed integration with Eclipse which is a bonus.) So, its goodbye to Trac, largely due to its poor administration interface.
Bugzilla sets the standard for Bug tracking applications. It allows a lot of customisation and is very mature at this stage. The only problem with Bugzilla, as with most web applications, is that to install it on your workstation it will require the installation of a lot of dependencies. Bugzilla will require the install of Apache, a mail transfer agent such as Sendmail and some extra Perl modules. Now I for one, don't feel comfortable having a full web server running on my machine just so I can track issues. Its an extra security risk and another application which requires updates. A safer option is to install Bugzilla within a virtual machine using Xen, VirtualBox or VMWare.
So that's the option I went for but, unfortunately, Bugzilla is a dog to install. Getting Bugzilla to play nicely with Apache is not easy. The standard packages (3.2) for both Ubuntu Server 8.10 and Debian 4.0 do not run OK after install. The forums are full of 'help' on which Apache and Bugzilla config files need to be updated. I won't get bogged down in the details - lets just say I wasted way too much time on this. (I haven't tested the install on Mandriva,CentOS or Fedora.) So its goodbye Bugzilla - life is too short.
The next application I looked at is Mantis. Its like a simplified version of Bugzilla. As with the others its a web app and requires Apache and MySQL. The install is less troublesome and it provides all the basic issue and bug tracking functionality required. It still has the downside of needing Apache etc. installed so the virtual machine route seemed a good move.
Then I remembered BitNami! BitNami provide open source application stacks which are self-contained. You can install the stack any place on your machine i.e. you are not installing an rpm with root privileges. Not only do they provide the basic LAMP stack, they also provide a large range of packaged applications. And yes, they package Mantis. The stack installer is excellent and the stack's directory can be moved around on your machine with no problems. To run the web server, there are start-up and shutdown scripts provided.
The advantage of BitNami is that everything: Apache, MySQL, PHPMyAdmin and Mantis, is contained within that one directory. Apache runs with the standard user's privileges and you can add links to the start-up and shutdown scripts to KDE's start and shutdown directories. Hey Presto - you have a locally installed version of Mantis with nearly zero configuration required.
BitNami seems like a great solution to me for developers who want LAMP based applications installed locally on their machine for testing and development. Its great advantage is that each install directory is self contained - you can install and run multiple stacks and configurations and it requires less maintenance than a VM.

Creative Zen with Mandriva 2008.0

2008-01-11T16:10:00.004+01:00

Recently got a Creative Zen 8GB and wanted to manage its music using Amarok. The PC runs Mandriva 2008.0 which comes with Amarok 1.4.8. The Creative Zen does not provide a standard USB mass storage interface so you can't just plug it into the USB port of your PC and start copying files. Creative devices use their own proprietary Media Transport Protocol so Amarok uses libmtp to access the device.
Unfortunately, the version of libmtp which comes with Mandriva 2008.0 is 0.2.1 which does not recognise the 8GB Zen (USB vid:0x41e pid:0x4157) so Amarok cannot connect to the Zen. However, the latest version of the libmtp code does recognise this Zen so the only solution was to rebuild Amarok using the latest libmtp.
Users who want to use their shiny new iPod may also face a similar issue. Mandriva 2008.0 comes with version 0.5.2 of libgpod, whereas the latest version of the code which supports new iPods such as the 80GB Classic is 0.6.0. You'll need a version of Amarok or gtkPod built against version 0.6 if you want to use a newer iPod.
I couldn't find adequate versions of these libraries on a site such as the PLF. Luckily the Mandiva wiki has quiet a good description of how to go about the whole process of rebuilding your packages see Mandriva RPM HowTo. I'm not going into the full details but heres an overview of the steps required:

Create an rpm directory in your home directory and create the required sub-directories as described in the HOWTO.

$ cd rpm
$ ls
BUILD/ RPMS/ SOURCES/ SPECS/ SRPMS/ tmp/

Rather than using the raw library source code to create the new rpms, a faster solution was to find more up-to-date rpms for a later version of Mandriva i.e. 2008.1 or Cooker and use these to build the new rpms for 2008.0. This is done by extracting the spec file and zipped source from the rpm and putting them in SPEC and SOURCES. Edit the spec file to build for 2008.0. When building, it will be built against the -devel libraries you have installed on your machine. Each rpm will have several dependencies so you'll have to install several -devel rpms before the new rpm is build successfully. These will include KDE devel libs, curl-devel, openssl-devel, musicbrainz-devel and many others.
Once all the new rpms are built, remove the old amarok, libmtp, libgpod etc. rpms and install your new rpms. I can confirm that the 8GB Zen works fine with libmtp version 0.2.5.

RPMS which I had to build:

$ cd SPECS
$ ls
amarok.spec gtkpod.spec gnomad2.spec libgpod.spec libmtp.spec

I can also confirm that the iPod Classic 80GB works with libgpod 0.6.0. Just remember that you must edit a text file on the actual iPod to get around Apple's stupid hash. README.sysinfo which comes with libgpod describes the process.

Setting Wakeup Time in BIOS

2007-05-11T14:14:00.001+02:00

I have MythTV 0.20 running on a PC which has an ASROCK K7S11 motherboard. After some googling and testing I was able to get the automatic wakeup for recording working. Included below is the script which is now working with this motherboard. If you have a different motherboard, be careful and don't use this script without understanding what you are doing. Got most of this online from other folks (sorry, can't remember who) and just needed to adapt the processing of the timestamp.

setwakeuptime.sh

#!/bin/bash

# set ACPI Wakup time in UTC format

stamp_file=/home/mythtv/timestamp
#echo $1\ $2 > $stamp_file

#echo $1\ $2 > /home/mythtv/setwakeuptime.inputs.log

# If using mythwelcome you can try the next line instead on the one above.
# echo >$stamp_file

# Read the date in the locale time format and add the time-zone info to the stamp_file
# datum=$(/bin/date -f $stamp_file +%F\ %T\ %z)
datum=$(/bin/date -u -d@$2 +%F\ %T)
echo $datum > $stamp_file

# reinterpret this in utc and write to alarm
# utcdatum=$(/bin/date -u -f $stamp_file +%F\ %T)
# echo $utcdatum > $stamp_file

#rm -f $stamp_file
echo $datum >/proc/acpi/alarm

exit 0

Hauppauge IR Remote with Linux

2007-04-18T21:56:00.000+02:00

I've recently built a home media center using MythTV. It has two DVB-T decoders, for which I'm using Hauppauge WinTV-Nova-T PCI cards. These cards work perfectly as they are supported by the linux dvb drivers. The cards come with an IR Remote, so to get this working you need to use LIRC and edit the LIRC config files. I've included my versions of both config files here since I had do some tricking about to get them working correctly. These are based on the config files which come with LIRC and MythTV with updates to support this remote. The remote serial number is A415-HPG-WE.

lircrc

begin
prog = mythtv
button = TV
repeat = 3
config = F1
end
begin
prog = mythtv
button = Videos
repeat = 3
config = F2
end
# Not yet defined
begin
prog = mythtv
button = Music
repeat = 3
config = F3
end
begin
prog = mythtv
button = Pictures
repeat = 3
config = F4
end
begin
prog = mythtv
button = Guide
repeat = 3
config = S
end
begin
prog = mythtv
button = Radio
repeat = 3
config = F6
end
begin
prog = mythtv
button = Up
repeat = 3
config = Up
end
begin
prog = mythtv
button = Down
repeat = 3
config = Down
end
begin
prog = mythtv
button = Left
repeat = 3
config = Left
end
begin
prog = mythtv
button = Right
repeat = 3
config = Right
end
# Channel Up
begin
prog = mythtv
button = ChannelUp
repeat = 3
config = Up
end
# Channel Down
begin
prog = mythtv
button = ChannelDown
repeat = 3
config = Down
end
# OK/Select
begin
prog = mythtv
button = OK
config = Space
end
# Play
begin
prog = mythtv
button = Play
config = Return
end
# Stop
begin
prog = mythtv
button = Stop
config = I
end
# Escape/Exit/Back
begin
prog = mythtv
button = Back
config = Esc
end
# Power Off/Exit
begin
prog = mythtv
button = Power
config = Esc
end
# Pause
begin
prog = mythtv
button = Pause
repeat = 3
config = P
end
# Mute
begin
prog = mythtv
button = Mute
repeat = 3
config = |
end
# Fast forward (30 sec default)
begin
prog = mythtv
button = NextTrack
repeat = 3
config = >
end
# Rewind (10 sec default)
begin
prog = mythtv
button = PreviousTrack
repeat = 3
config = <
end
# Skip forward (10 min default)
begin
prog = mythtv
button = Skip
repeat = 3
config = End
end
# Skip backward (10 min default)
begin
prog = mythtv
button = Replay
repeat = 3
config = Home
end
# Record
begin
prog = mythtv
button = Record
repeat = 3
config = R
end
# Delete
begin
prog = mythtv
button = Red
repeat = 3
config = D
end
# Decrease play speed
begin
prog = mythtv
button = Green
repeat = 3
config = J
end
# Display EPG while in live TV,
# View selected show while in EPG
begin
prog = mythtv
button = Menu
repeat = 3
config = I
end
# Scroll up
begin
prog = mythtv
button = VolumeUp
repeat = 3
config = F11
end
# Scroll down
begin
prog = mythtv
button = VolumeDown
repeat = 3
config = F10
end
# Bring up OSD info
begin
prog = mythtv
button = Go
repeat = 3
config = O
end
# Change display aspect ratio
begin
prog = mythtv
button = PreviousChannel
repeat = 3
config = W
end
# double speed watch
begin
prog = mythtv
button = Yellow
repeat = 3
config = J
end
# change tuners
begin
prog = mythtv
button = Subtitles
repeat = 3
config = Y
end
# Bring up Time stretch
begin
prog = mythtv
button = Blue
repeat = 3
config = A
end
# Numbers 0-9
begin
prog = mythtv
button = 0
repeat = 3
config = 0
end
begin
prog = mythtv
button = 1
repeat = 3
config = 1
end
begin
prog = mythtv
button = 2
repeat = 3
config = 2
end
begin
prog = mythtv
button = 3
repeat = 3
config = 3
end
begin
prog = mythtv
button = 4
repeat = 3
config = 4
end
begin
prog = mythtv
button = 5
repeat = 3
config = 5
end
begin
prog = mythtv
button = 6
repeat = 3
config = 6
end
begin
prog = mythtv
button = 7
repeat = 3
config = 7
end
begin
prog = mythtv
button = 8
repeat = 3
config = 8
end
begin
prog = mythtv
button = 9
repeat = 3
config = 9
end
### MPlayer lirc setup
# Show OSD
begin
prog = mplayer
button = Menu
repeat = 3
config = osd
end
# Pause playback
begin
prog = mplayer
button = Pause
repeat = 3
config = pause
end
# Skip ahead a minute if playing
# If paused, resume playing
begin
prog = mplayer
button = Play
repeat = 3
config = seek +1
end
# Stop playback and exit
begin
prog = mplayer
button = Stop
repeat = 3
config = quit
end
# Mute
begin
prog = mplayer
button = Mute
repeat = 3
config = mute
end
# Seek back 10 seconds
begin
prog = mplayer
button = FRW
repeat = 3
config = seek -10
end
# Seek forward 30 seconds
begin
prog = mplayer
button = FFW
repeat = 3
config = seek +30
end
# Quit
begin
prog = mplayer
button = Back
repeat = 3
config = quit
end
# Seek forward 10 minutes
begin
prog = mplayer
button = Skip-For
repeat = 3
config = seek +600
end
# Seek backward 10 minutes
begin
prog = mplayer
button = Skip-Back
repeat = 3
config = seek -600
end
# Toggle full-screen
begin
prog = mplayer
button = FULL
repeat = 3
config = vo_fullscreen
end
### Xine lirc setup
begin
prog = xine
button = Play
repeat = 3
config = Play
end
begin
prog = xine
button = Stop
repeat = 3
config = Stop
end
begin
prog = xine
button = Power
repeat = 3
config = Quit
end
begin
prog = xine
button = Pause
repeat = 3
config = Pause
end
begin
prog = xine
button = CH+
repeat = 3
config = EventUp
end
begin
prog = xine
button = CH-
repeat = 3
config = EventDown
end
begin
prog = xine
button = Vol-
repeat = 3
config = EventLeft
end
begin
prog = xine
button = Vol+
repeat = 3
config = EventRight
end
begin
prog = xine
button = OK
repeat = 3
config = EventSelect
end
begin
prog = xine
button = Back
repeat = 3
config = Menu
end
begin
prog = xine
button = FFW
repeat = 3
#config = SpeedFaster
config = SeekRelative+60
end
begin
prog = xine
button = FRW
repeat = 3
#config = SpeedSlower
config = SeekRelative-60
end
begin
prog = xine
button = FULL
repeat = 3
config = Volume+
end
begin
prog = xine
button = BLANK
repeat = 3
config = Volume-
end
begin
prog = xine
button = Mute
repeat = 3
config = Mute
end
begin
prog = xine
button = Menu
repeat = 3
config = RootMenu
end
begin
prog = xine
button = SKIP
repeat = 3
config = EventNext
end
begin
prog = xine
button = REPLAY
repeat = 3
config = EventPrior
end
begin
prog = xine
button = Go
repeat = 3
config = OSDStreamInfos
end
begin
prog = xine
button = Red
repeat = 3
config = Quit
end

and heres my version of /etc/lircd.conf

begin remote

name hauppauge_nova_t_uk
bits 16
eps 30
aeps 100

one 0 0
zero 0 0
pre_data_bits 16
pre_data 0x8001
gap 135803
toggle_bit 0

begin codes
Power 0x0074
Go 0x0161
TV 0x0179
Videos 0x0189
Music 0x0188
Pictures 0x016F
Guide 0x016D
Radio 0x0181
Up 0x0067
Left 0x0069
Right 0x006A
Down 0x006C
OK 0x001C
Back 0x00AE
Menu 0x008B
VolumeUp 0x0073
VolumeDown 0x0072
PreviousChannel 0x019C
Mute 0x0071
ChannelUp 0x0192
ChannelDown 0x0193
Record 0x00A7
Stop 0x0080
Replay 0x00A8
Play 0x00CF
Skip 0x00D0
PreviousTrack 0x00A5
Pause 0x0077
NextTrack 0x00A3
1 0x0002
2 0x0003
3 0x0004
4 0x0005
5 0x0006
6 0x0007
7 0x0008
8 0x0009
9 0x000A
0 0x000B
Text 0x0184
Subtitles 0x0172
Red 0x018E
Green 0x018F
Yellow 0x0190
Blue 0x0191
end codes

end remote

A Quick Overview of Character Encoding Schemes

2006-03-01T20:01:00.012+01:00

Introduction

Most writing systems are based on the representation of consonants, vowels, syllables or meanings using graphical characters. Alphabets such as the Latin or Greek alphabets represent vowels and consonants by characters. Abjads such as Arabic or Hebrew normally represent only consonants by characters. Syllabic alphabets such as Bengali represent each syllable (a consonant and vowel combination) by a character. More complex scripts such as Japanese Kanji use characters that represent both a syllable and meaning.

Alphabets such as that used by the English language have a small number of characters in the range of 20 to 50 characters. However in this English-centric world of computing, it is easy to forget that complex scripts such as Chinese can have up to 10,000 different characters. Technology must therefore support character sets far larger than the small set the English alphabet requires.

A character set encoding allows the representation of a graphical character in a numerical format so that it may be manipulated and stored using software or hardware. We need to map each character to a numerical code that can then be converted to binary and represented as one or several bytes.

A single byte, with its 8 bits, can represent 256 values ranging from zero (all bits set to zero) to 255 (all bits set to 1). It is therefore apparent that using a single byte to represent each character in a complex script is not feasible several bytes are required. Various character encoding schemes have been developed that map a large range of characters to multiple byte sequences and some of these are now examined.

Comparison

Rather than examining the representation of every character for every encoding scheme, two representative characters are chosen and compared for each character encoding scheme. The characters used are the Latin alphabet's A and the Arabic character ۸, which is the number 8 . Since it all comes down to bits and bytes in the end, the byte values used to represent each character are represented in hexadecimal format.

ASCII

ASCII was first published as a standard by the American Standards Association in 1963. The latest version was published by ANSI (American National Standards Institute) in 1986 as ANSI specification X3.4-1986. ASCII characters are represented by 7 bits although extended versions are available that use the 8th bit to represent a larger range of characters. ASCII can therefore support a maximum of 128 characters which falls far short of the requirement to support large character sets. The ASCII specification is not freely available and must be purchased from ANSI.

Key features;

Characters represented by 7-bit code

Includes 95 printable characters

Includes non-printable character codes

Table 1: ASCII Example
Character	Description	Representation in Bytes
A	Latin A	0x41
۸	not supported	not supported

ISO/IEC 8859

The ISO/IEC developed this character encoding specification that represents a large range of characters using 8 bits. The specification is divided into 15 different parts, each of which deals with a specific group of characters. ISO/IEC 8859-1, also known as Latin1, represents the most common Latin alphabet characters. Final versions of the specification must be purchased from the ISO, however draft versions are available from the W3C and other sites online.

Features;

Each character represented by 8 bits i.e. 1 byte

15 parts to the specification

Each part represents up to 256 characters

Some characters are present in more than one part of the specification

Table 2: ISO/IEC 8859 Example
Character	Description	Representation in Bytes
A	ISO 8859-1 (Latin1)	0x41
	ISO 8859-2 (Latin2)	0x41
	ISO 8859-3 (Latin3)	0x41
	ISO 8859-6 (Arabic)	0x41
۸	ISO 8859-6 (Arabic)	not supported

Table 2 demonstrates that multiple tables have codes for a core set of characters. The table also demonstrates that part 6 does not support all Arabic characters as there is no representation for the character۸. As all characters must be stored in 1 byte and each part of the specification contains a core set of characters, the actual range of complex characters supported is small. Also, as each part uses the same character codes for different characters, the part of the specification that text uses must be known for the correct interpretation of characters.

Unicode

The Unicode Consortium, a U.S-based industry group, published a first version of this standard in 1991 and continues to develop the standard today. It is closely related to the ISO/IEC 10646 specification as the two organisations aim to keep the standards synchronised. Each character is represented by a uniquecode pointand when writing a code point it is normal to precede the code point number by U+ e.g. U+0041.

The Universal Character Set (UCS) contains all possible characters and this large set is divided into planes, each of which contains 65,536 characters (the maximum number that can be represented by 2 bytes). The Basic Multilingual Plane (BMP) is the first plane and it contains most of the commonly used characters.

Two types of encodings are available for Unicode characters. They can be encoded using the Unicode Transformation Format (UTF) format or using the UCS format, defined by ISO/IEC 10646.

Table 3: Unicode Code Points
Character	Code Point
A	U+0041
۸	U+06F8

UTF-8

This is a variable length encoding scheme that uses one to four bytes to encode each code point. RFC 3629 defines the UTF-8 encoding and it is also described in the Unicode specification. Code points from U+0000 to U+007F map directly to the ASCII character set therefore ASCII character codes form correct UTF-8 characters. Table 4 shows the number of bytes needed to encode a character depending on its code point.

Features;

Variable length encoding

Code points encoded using from 1 to 4 bytes

Table 4: UTF-8 Byte Requirements
Code Point Range	Bytes Used
U+0000 – U+007F	1
U+0080 – U+07FF	2
U+0800 – U+0FFF	3
U+10000 –	4

Table 5: UTF-8 Example
Character	Description	Representation in Bytes
A	Code point U+0041	0x41
۸	Code point U+06F8	0xDB 0xB8

UTF-16

This is a variable length encoding scheme that encodes code points as 16-bit code values. If a character is not in the BMP i.e. its code point is greater than U+FFFF, then two or more code values are used to represent the character. RFC 2781 defines the UTF-16 encoding and it is also described in the Unicode specification. Windows NT, Java and Qt use UTF-16 to internally represent characters.

As each code value requires 2 bytes, the endianess of the bytes must be communicated so that the code values are interpreted correctly. For this reason, the Byte Order Mark (BOM) is used. The code point U+FEFF is appended before the UTF-16 data. Therefore a byte sequence 0xFE 0xFF indicates the data is in big-endian order.

Features;

Variable length encoding scheme

A code point is represented by one or more two-byte code values

Table 6: UTF-16 Example
Character	Description	Representation in Bytes
A	Code point U+0041	0x41
۸	Code point U+06F8	0x06 0xF8

Base64

Base64 is not a character encoding scheme but is nonetheless interesting. Base64 encoding is used to transform a sequence of byte values (they may be image information, program data or anything else) to a textual representation. In Base 64 the byte values are transformed to a subset of the ASCII printable characters. The character sequence may then be transmitted by a protocol that only accepts ASCII encoding and later it can be transformed to its original format. RFC 2045 describes Base64 encoding as used in the MIME email format.

Conclusion

Several of the most common character encoding schemes have been briefly described. Each scheme has its own advantages and disadvantages but all have a similar aim, which they achieve to varying degrees of success. Unicode is the most comprehensive specification and it continues to be developed, adding support for new character sets. Multiple other character encoding exist, however Unicode and ISO 10646 are the prevalent standards today.

References

http://www.omniglot.com/index.htm provides an informative overview of alphabets and complex writing systems.

http://www.ietf.org/rfc/rfc1345.txt RFC 1345 provides standardised names for characters that should be used in other Internet protocol definitions.

http://www.ietf.org/rfc/rfc3629.txt RFC 3629 describes the UTF-8 encoding scheme.

http://www.ietf.org/rfc/rfc2781.txt RFC 2781 describes the UTF-16 encoding scheme.

http://www.ietf.org/rfc/rfc2045.txt RFC 2045 describes the MIME email format and Base64 encoding.

http://en.wikipedia.org/wiki/Character_encoding Wikipedia provides an extensive discussion of many character encoding schemes and standards.

Author Information

Michael McCarthy is a professional Software Engineer who specialises in Java and open source software development. For more information on his activities, visit http://www.mickmccarthy.com .

No more CMS....back to static html

2006-01-26T10:22:00.001+01:00

After tiring of a second CMS, the time came to rethink the website and now you can see the results.

A Content Management System is great - you can add new text and pages using your web browser and all the site layout is already there. The problem with CMSs is that to make the site interactive, a lot of functionality must be added which exposes a lot more functionality that can be exploited. Before Christmas I was using b2evolution (v0.9.0.11), a blogging CMS, which I had to remove because of a constant bombardment of robots trying to add links for tracebacks to the site. I believe this vulnerability was fixed in later versions but I used the opportunity to try e107 (v0.6172). e107 is all full featured CMS, built with PHP like b2evolution, that has lots of functionality. However, it has vulberabilities of course and these were expoited before I could upgrade to the latest version.

To me, the problem seems to be that using a CMS requires almost constant upgrading to the very latest version. This is ok if you want to upgrade several hours a week to a website but not ok if the website is a background task. So I've decided to go from using a CMS i.e. GNU/Linux, Apache, PHP and mySQL to using plain old XHTML and CSS i.e. GNU/Linux and Apache. This will definitely reduce the attack vectors and hopefully reduce the attacks. Time will tell and remembering that nothing is 100% secure.

There is a great site called Open Source Web Design where web designers showcase their templates and designs - its like Sourceforge for designers. I found this design that was put together by a guy caled Andreas Viklund. Its pure XHTML and CSS and W3C compliant.

On a last note, if you are interested in trying out some CMSs, this site is great - it lets you try out the CMS without downloading anything.

FRET v0.0.6 - here at last

2006-01-19T10:58:00.001+01:00

Arriving some weeks later than planned, the new version includes new functionality, several bug fixes and refactored code.

I've added two new commandine options: --phase and --scan, that allow the user to specify a particular range of tests to run. Several bugs have been fixed and the Group code (and concept) is now gone - this makes the code size more manageable and hasn't removed any currently used functionality.

I also had a chance to analyse the performance of FRET for the first time. As expected, the ScanGrind analysis (it looks for data that is common to two of more files) is the major contributor. Its analysis duration grows exponentially with file size. This can unfortunately lead to very long durations for files over 1 MB - I estimated that it takes 27 hours to analyse two 1.7 MB files. This is not unexpected although I am sure there is room for improvement in the algorithm. However, currently the algorithm is totally brute force - it compares the start of one file with the end of another etc. and I don't think that this makes sense when examining file formats. A more intelligent approach would be to divide a file into zones, maybe based on already detected Grams, and then to look for commonalities between those zones.

FRET v0.0.5 Released

2005-09-14T23:58:00.001+02:00

Quoting from the FRET website, " Wednesday 14 September 2005. The worlds press gathered today in front of a small Unix terminal to watch the release of v0.0.5 of FRET into the wild. The tarball was reticent at first, looked back once at the gathered crowd and then swiftly ran away to join its own kind."

Actually, its a bug-fix release with one significant change - the improved ScanGrind function. (As a side note, I'm thinking of spinning off the ScanGrind function as a self-contained C module for other developers to use, maybe even in the Public Domain). Now I need to decide what will go into release 0.0.6. The three main options I am considering at the moment are: - Add a test framework for libfret, probably using CUnit - Add a way of measuring FRET's functionality i.e. how much of a set of test files does it correctly analyse. - Make libfret thread-safe I've also noticed that FRETs been added to the FileFormat site in the tools section - always good to see some interest!

FRET v0.0.5 Design Changes

2005-09-02T17:57:00.010+02:00

Currently work is progressing on FRET version 0.0.5 and the major change for this version is a swap in the order of the Phase 4 and Phase 5 Scans. An improvement in ScanGrind was the driving force for this change. Previously, when a data structure, which was common to two Buffers, was identified by ScanGrind, a Group Gram was created with the offset from the first Buffer. This caused the loss of valuable information about the structures location in the second Buffer. The new version of ScanGrind now creates two Buffer Grams, one for each Buffer and no Group Gram is created. A Phase 5 Scan will then notice that two Buffer Grams are identical and will create a Group Gram with frequency 2.

Order of analysis phases modified

Detecting data structures and formats in a buffer or file is a complex task. For this reason the process has been sub-divided into multiple sub-tasks, each of which performs a small piece of analysis. It is through the amalgamation of these functions that a clear picture is derived of the structures within the target data. FRET divides these tasks into 6 Phases. Each small task that performs a specific function in the analysis is called a Scan. Each Phase contains multiple Scans and all the Scans within a Phase share the same characteristics (and interface). This modular design allows for the easy addition, deletion or re-ordering of Scans. As new Scans are added and refined, the program will grow in ability. Each phase of analysis is now briefly described.

Before commencing the description of the phases, it is key to remember the aim of FRET. It is designed as a commandline utility (and library) that takes a number of similar raw data files (or buffers), analyses them and outputs a table describing the layout of data in these files. If all the files are different then it will find little in common between the files. However, if the data follows the same format i.e. values change but the layout is the same, then a lot can be learned from the analysis and a good guess can be taken at the file format. In FRET terminology a detected pattern of bytes in a file is called a Gram, a term derived from the Greek for a letter, which is commonly used in cryptanalysis.

The first step in the analysis of a buffer are the Phase 1 Scans. These Scans examine the raw data, identify if compression or obfuscation is used and transform the data to its original order. If the later Phases were to try and analyse compressed data, for example, then it would be like trying to analyse random data for patterns - some would be found but they would be meaningless. Currently, there are no Phase 1 Scans implemented in FRET. It is probable that the first Scan to be added will be to uncompress a buffer.

Phase 2 Scans look at each file individually and attempt to identify data structures that are file format independent. These include ASCII strings, Unicode strings, fill bytes, x86 instructions and limitless other recognisable data formats. Depending on the Scan and raw data, there is a varying level of confidence that a detected Gram is valid, it may be a false positive, therefore each Gram is assigned a risk. The risk of a Gram is the probability that it is a randomly occurring pattern in the data and is not valid. This allows for the later sanity checking of Grams.

Once a Buffer of data has been analysed and Grams have been detected, it may be possible to use these Grams to detect further patterns in the data. Phase 3 Scans compare detected Grams against the raw data looking for relationships. One example of this is an offset to data in a file - often within a file format there is a table that contains identifiers and offsets to data later in the file. For example, if a string is detected in the file and earlier in the file a value that points to the start of this string is detected, it is possible that this is an offset. It may also be possible to detect fields that describe the length of other fields in a file.
At this stage, no more information can be gained from looking at a file in isolation. It is now time to compare it to its peers and this is where the real power of FRET is unleashed. Phase 4 Scans compare the raw data in multiple files, looking for commonalities and differences. Historically binary diff tools have been used for this type of analysis and they remain a valuable tool, yet there is value in focusing on the similarities between files, not the differences. scanGrind compares two files looking for all the byte patterns that are the same and they don't have to be in the same place in both files.

Now you may think it is finished but no, there are two more phases of analysis left. Firstly, the Phase 5 Scans compare the Grams that have been detected for each Buffer and create new Group Grams. This analysis needs to be fuzzy - a file format may specify a string at a certain location but the length and exact offset of the string will be different between the two files. The algorithms for the Phase 5 scans must compare Grams, looking at factors such as 'approximate' location, preceding Grams, Gram type etc. and take a guess.

Finally, once all this is completed, it is time to clean-up the results. Phase 6 Scans iterate through the detected Grams, removing or amalgamating Grams based on a range of criteria. After Phase 6, the Grams for the Buffers and Group can be passed to the user who can further process the data using custom scripts or tools.

Using the Olympus C-770 with GNU/Linux

2005-08-02T12:21:00.003+02:00

The Camedia C-770 Ultra Zoom was released in early 2004 by Olympus as the top-of-the-range camera in their UZ series. It features a 10x optical zoom with 4 Megapixel sensor, however the distinguishing feature of this camera is its ability to record 640x480 video at 30fps in MPEG-4 format. This allows the camera to record 30 minutes of TV quality video on a 512MB xD card. Olympus only includes Windows and Mac applications with the camera, therefore GNU/Linux or other OS users must find other software applications to utilise the camera. Olympus Camedia Master 4.2 for Windows and Macintosh provides the ability to upload, organise and print your photos. Also included is VideoStudio 7 (SE Basic) which provides video editing capabilities.

Connecting to your PC

The camera supports both the USB Mass Storage and PTP protocols therefore connecting the camera to a PC should not be a problem. In fact, if you use a modern distribution (such as Mandriva 2005 LE) you should not have to change your PC's configuration before uploading your photos and videos.

It is advisable to connect the camera to the recommended DC power source before uploading your photos since otherwise you will drain your batteries quickly. Connect the camera to the PC using the supplied USB cable and power-on the camera. Select the PC option and the camera will connect to PC. If you use an up-to-date desktop then an icon should appear that identifies the camera. If no icon appeared on the desktop then configure the desktop to display icons for newly mounted storage devices. However it may be that the OS has not mounted the device correctly. Check the directory /mnt/ to see if the camera is listed - if it is not listed then the OS requires correct configuration. This configuration is outside the scope of this document, however there is a lot of good documentation describing how to do this.

The Linux Documentation Project provides a HOWTO on connecting USB cameras to GNU/Linux PCs. The guide is a little out-dated but will point you in the right direction. The HOWTO is available at http://www.tldp.org/HOWTO/USB-Digital-Camera-HOWTO/index.html. A better option is to investigate web forums for your particular distribution - it is guaranteed that several users have solved the same problem already.

Photo Manipulation

Once the camera is correctly connected, photos can be uploaded to the PC using a variety of the methods. The most basic method is to open the directory on the camera that contains the photo and video files and to copy these to a directory on the PC. This works perfectly well but can become boring after a while so it is recommended to automate this process as much as possible. Use a photo album application to organise and automate the upload of your photos. One example of such an application is Digikam, the KDE photo application. Digikam allows the definition of one or multiple cameras and it then handles the upload of photos for these cameras and their organisation into albums. More information of Digikam is available at http://www.digikam.org/ and there should be a packaged version available for your distribution.

MPEG-4 Movies

The C-770 stores video in the MPEG-4 format and the soundtrack is stored in the AAC audio format. Both the audio and video streams are then packaged in the MP4 file format, which is essentially a Quicktime file format. As of the time of writing, common GNU/Linux media players such as Kaffeine may or may not play these files, depending on the version of the application you have. All of the software is available, however due to patent and licensing concerns it is not included as default in some distributions. Playing and editing MPEG-4 videos is possible, it justs requires a little know-how. Currently there are two options and these are now examined.

The first option is to acquire application software that will allow you to play and edit the MP4 files. The best option for playing the movies is Xine, a movie player that includes MPEG-4 support and which can use FAAD2 to provided AAC decoding. A FAAD2 enabled version of Xine is required therefore some investigation may be needed. As for the editing of MPEG-4 video and audio, I am not aware of a satisfactory application - the best option is to transcode the movies to a more editable format such as MPEG-2 (see below).

The second option is to convert these MP4 files to a more commonly supported format that will allow you to play, share and edit these movies. Converting the format of an audio or video stream is called transcoding and several tools are available to accomplish this task. FAAD2 is a decoder for AAC streams that can transcode them to other audio formats such as mp2, mp3 or AC3. FFMPEG is a powerful video decoder that supports the MPEG-4 video format. FFMPEG also includes a basic movie player called ffplay but this does not currently (version 0.4.9) support the AAC format. MENCODER can also be used to transcode the video from MPEG-4 to a range of other formats. The next decision to make is to decide the target file format to use. The most common GNU/Linux movie players such as Kaffeine and MPlayer support several movie formats such as mpg, avi and divx so if playing the movies on a GNU/Linux PC is the aim then any of these formats will suffice. If care is taken during the transcoding process then the movie quality will not be degraded but the file size may increase depending on the file format you use. If the aim is to create DVDs from the mp4 movies, then transcode the movies to either an MPEG-2 VOB format or a DivX format and use a tool such as QDVDAuthor to create the DVD. The movies can be edited and amalgamated with a tool such as Kdenlive or LVE if they are in the MPEG-2 format.

As with all GNU/Linux software, the choices are almost limitless and can be confusing. To simplify the process for users of the C-770 and other mp4 capable cameras, I have created a Perl script that automates the transcoding of the mp4 files to more usable formats.

mp4conv

mp4conv.pl is a perl script that uses the FAAD2, FFMPEG and MPEG2ENC tools to transcode mp4 file to other common formats. Download the script and place it in a directory that is on your PATH so that you can execute it. Also, don't forget to chmod +x it to make the script executable. The command mp4conv --help will then print the usage options of the script. It currently supports the following operations;

MPEG2: --mpeg2 transcodes the video and audio to plain old MPEG2 at the same quality and packages then in an mpg file. Expect a much larger file that can be viewed almost anywhere.
MPEG4. --mpeg4 doesn´t touch the video, transcodes the audio to MPEG2 and stores the movie in the avi format. This makes the movie more viewable but keeps the same level of compression.
DVD. --dvd creates a PAL compatible MPEG2 stream, with MP2 soundtrack and DVD NAV packets that you can then use to create DVDs. Supporting NTSC only requires changing the image dimensions and fps within the script.
WINDOWS. --windows creates an avi file that all your sad Windows-using friends can view.
DV. --dv is supposed to create a DVI file that you can then edit. This option is currently not completed.

The power of this script is that with one command you can convert all your mp4 files to a range or common movie formats - no more pointing and clicking for hours!

Conclusion

The C-770 is a good camera with impressive movie recording abilities but some effort is required to use it to full effect with current GNU/Linux distributions. Use the following references to investigate the options and utilize the mp4conv.pl script to automate the drudgery. Happy snapping!

References

http://linux.omnipotent.net/article.php?article_id=11500 gives a basic introduction to the AAC audio format.

http://www.ndeepak.info/tech/c770uz.php provides a guide to connecting the camera to an older Debian PC. Debian Sarge may provide better support out of the box - I'm not sure.

http://www.audiocoding.com is home of the FAAD2 audio decoder.

http://ffmpeg.sourceforge.net/ is the home of FFMPEG, the video decoder.

http://mjpeg.sourceforge.net/ is home to the excellent MJPEG toolset, which provides mpeg2enc.

http://xinehq.de/ is the HQ of Xine, a powerful media player.

The Script

#! /usr/bin/perl -w

# Olympus C-770 MPEG-4 Movie Conversion Script
#
# M.McCarthy
# Version 0.6

use strict 'vars';
use English;

# constants
my $SUCCESS = 0;
my $FAIL = 77;

# Video formats
my $MPEG2       = "MPEG2 MPG";
my $MPEG2EXT    = "_mpeg2";
my $DVD         = "DVD AVI";
my $DVDEXT      = "_dvd";
my $MPEG4       = "MPEG4 AVI";
my $MPEG4EXT    = "_mpeg4";
my $WINMV       = "WINDOWS AVI";
my $WINMVEXT    = "_wmv";
my $DV          = "DV";
my $DVEXT       = "_dv";
my $YOUTUBE     = "YOUTUBE PREFERRED";
my $YOUTUBEEXT = "_youtube";

my $VERSION = "0.6";

# function prototypes
sub mp4convert( $$ );
sub printHelp();

#### Main Function starts here ####

# print header info
print "*************************************************\n";
print "* Olympus C-770 Movie Conversion Utility v.".$VERSION." *\n";
print "*    (c) M.McCarthy 2005,2006                   *\n";
print "*************************************************\n";

# check dependencies
my @clargsFaad = ("which", "faad");
system(@clargsFaad) == 0
    or die "Error: FAAD2 not available";
my @clargsMencoder = ("which", "ffmpeg");
system(@clargsMencoder) == 0
    or die "Error: FFMPEG not available";
@clargsMencoder = ("which", "mpeg2enc");
system(@clargsMencoder) == 0
    or die "Error: mpeg2enc not available";

my $file;

# set defaults
my $outputFormat = $MPEG4; # default is MPEG-4 AVI with sound
# used to generate statistics at end
my $numOfFiles = 0;
my $numOfConversions = 0;

not @ARGV
    and die "Error: No filename passed as argument";

# parse the commandline arguements
for $file ( @ARGV )
{
    if ( -e $file
        and $file =~ m/.+mp4$/ )
    {
        $numOfFiles= $numOfFiles+1;
    
        if( mp4convert( $file, $outputFormat ) != 0 )
        {
            warn "Error: $file could not be converted";
        }
        else
        {
            $numOfConversions = $numOfConversions + 1;
        }
    }
    elsif( $file =~ m/.*(--mpeg2).*/ )
    {
        $outputFormat = $MPEG2; 
    }
    elsif( $file =~ m/.*(--dvd).*/ )
    {
        $outputFormat = $DVD;
    }
    elsif( $file =~ m/.*(--mpeg4).*/ )
    {
        $outputFormat = $MPEG4;
    }
    elsif( $file =~ m/.*(--windows).*/ )
    {
        $outputFormat = $WINMV;
    }
    elsif( $file =~ m/.*(--dv).*/ )
    {
        $outputFormat = $DV;
    }
    elsif( $file =~ m/.*(--youtube).*/ )
    {
        $outputFormat = $YOUTUBE;
    }
    elsif( $file =~ m/.*(--help).*/ )
    {
        printHelp();
    }
    else 
    {
        warn "Warning: $file is not a valid option or filename";
    }
}

print "**********************************************\n";
print " Summary:\n";
print "     Files Processed : ", $numOfFiles, "\n";
print "     Successful      : ", $numOfConversions, "\n";
print "     Errors          : ", ($numOfFiles-$numOfConversions), "\n";
print "**********************************************\n";

exit;

# passed an mp4 filename, converts to avi format
sub mp4convert( $$ )
{
    my $status = $SUCCESS;

    my $srcFile = shift( @_ );
    print "Source File: ", $srcFile, "\n";

    my $output = shift( @_);
    print "Target Format: ", $output, "\n";
    
    # generate temp audio filename
    $_ = $srcFile;
    s/(mp4)$/wav/;
    my $audioFile = $_;
    print "==> Temp audio file: ", $audioFile, "\n";

    # extract audio from mp4 file
    my $faadArgs = "faad -o ".$audioFile." ".$srcFile;
    print "==> ", $faadArgs, "\n";
    system($faadArgs) == 0
        or warn "Error: Could not extract audio from MPEG-4 file";
    if( -e $audioFile )
    {
        print "OK: Audio extraction completed.\n";
    }
    else
    { 
        warn "Error: No temp audio file produced";
        return $FAIL;
    }

    # setup output options for conversion
    my $outOpts;
    my $nameExt;
    
    if( $output eq $MPEG2 )
    {
        $outOpts = "-sameq -s 640x480 -aspect 4:3 -vcodec mpeg2video -acodec mp2";
        $nameExt = $MPEG2EXT.".mpg";
    }
    elsif( $output eq $DVD )
    {
        $outOpts = "-b 2000 -s 720x576 -r 25 -an -f yuv4mpegpipe";
        # $outOpts = "-sameq -target pal-dvd -an";
        $nameExt = $DVDEXT.".yuv";
    }
    elsif( $output eq $MPEG4 )
    {
        $outOpts = "-sameq -s 640x480 -vcodec mpeg4 -acodec mp2";
        $nameExt = $MPEG4EXT.".avi";
    }
    elsif( $output eq $WINMV )
    {
        $outOpts = "-sameq -s 640x480 -vcodec wmv2 -acodec mp2";
        $nameExt = $WINMVEXT.".avi";
    }
    elsif( $output eq $DV )
    {
        $outOpts = "-sameq -s 720x576 -vcodec dvvideo";
        $nameExt = $DVEXT.".dv";
    }
    elsif( $output eq $YOUTUBE )
    {
 $outOpts = "-sameq -s 320x240 -vcodec mpeg4 -acodec mp3";
 $nameExt = $YOUTUBEEXT.".mpg";
    }
    else
    {
        warn "Error: No output option selected";
        $status = $FAIL;
    }

    # generate target filename
    $_ = $srcFile;
    s/(\.mp4)$/$nameExt/;
    my $outFile = $_;
    print "Target File: ", $outFile, "\n";    
   
    # generate the target file
    my $mencArgs = "ffmpeg -y -i ".$srcFile." -i ".$audioFile." ".$outOpts." ".$outFile;
    print "==> ", $mencArgs, "\n";
    
    if( $status == $SUCCESS
        && system($mencArgs) != 0 )
    {
        warn "Error: could not convert $srcFile";
        $status = $FAIL;
    }
    else
    {
        print "OK: Video conversion successful.\n";
    }
    
    # if dvd source required, then generate a DVD standard stream
    
    # generate temp mp2 filename
    $_ = $audioFile;
    s/(\.wav)$/_tmp\.ac3/;
    my $m2aFile = $_;
    
    # convert pcm file to MPEG-2 audio
    if( $output eq $DVD 
        && $status == $SUCCESS )
    {
 # print "AC3 Audio File: ", $m2aFile, "\n";
        $mencArgs = "ffmpeg -i ".$audioFile." -acodec ac3 -ab 128 ".$m2aFile;
        print "==> ", $mencArgs, "\n";
        
        if( system($mencArgs) != 0 )
        {
            warn "Error: could not convert $audioFile to AC3 format";
            $status = $FAIL;
        }
        else
        {
            print "OK: AC3 Audio conversion successful.\n";
        }
    }
        
    # generate temp mpeg-2 filename
    $_ = $srcFile;
    s/(\.mp4)$/_tmp\.mpg/;
    my $m2vFile = $_;
    
    # convert MPEG-2 video to DVD standard MPEG-2 video
    if( $output eq $DVD 
        && $status == $SUCCESS )
    {
        
        # print "MPEG-2 Video File: ", $m2vFile, "\n";
        
        $mencArgs = "mpeg2enc -n p -f 8 -s -o ".$m2vFile." < ".$outFile;
        print "==> ", $mencArgs, "\n";
        
        if( system($mencArgs) != 0 )
        {
            warn "Error: could not convert $outFile to MPEG-2 DVD video format";
            $status = $FAIL;
        }
        else
        {
            print "OK: MPEG-2 DVD video conversion successful.\n";
        }
    
    }
        
    # multiplex MPEG-2 video, MP2 audio and DVD NAV packets
    if( $output eq $DVD 
        && $status == $SUCCESS )
    {
        # multiplex the files together
        $_ = $outFile;
        s/dvd\.yuv/dvdvob\.mpg/;
        my $dvdFile = $_;
        print "Target File: ", $outFile, "\n";
                
        $mencArgs = "mplex -f 8 -o ".$dvdFile." ".$m2vFile." ".$m2aFile;
        print "==> ", $mencArgs, "\n";
        
        if( system($mencArgs) != 0 )
        {
            warn "Error: could not create DVD VOB file";
            $status = $FAIL;
        }
        else
        {
            print "OK: DVD VOB file creation was successful.\n";
        }
    }
    
    
    #cleanup temporary files
    if ( -e $audioFile )
    {
        system("rm -f $audioFile") == 0
            or warn "Warning: Could not remove temp file $audioFile";
    }
    if ( -e $m2aFile )
    {
        system("rm -f $m2aFile") == 0
            or warn "Warning: Could not remove temp file $m2aFile";
    }
    if ( -e $m2vFile )
    {
        system("rm -f $m2vFile") == 0
            or warn "Warning: Could not remove temp file $m2vFile";
    }
    if ( $output eq $DVD
         && -e $outFile )
    {
        system("rm -f $outFile") == 0
            or warn "Warning: Could not remove temp file $outFile";
    }

    return $status;
} 


sub printHelp()
{
    print "\n";
    print $PROGRAM_NAME, "  input filename(s)\n";
    print "    --help\n";
    print "           Print this screen\n";
    print "    --mpeg2\n";
    print "           MPEG-2 Video with MP2 Audio\n";
    print "    --divx\n";
    print "           MPEG-4 DIVX-5 Video with AC-3 Audio\n";
    print "    --mpeg4 (default)\n";
    print "           MPEG-4 Video with MP2 Audio\n";
    print "    --windows\n";
    print "           WMV Video with MP2Audio\n";
    print "    --dv\n";
    print "           Digital Video DV\n";
    print "    --youtube\n";
    print "           Youtube Preferred Format\n";
    return;
}

FRET version 0.0.4 released!

2005-07-22T10:55:00.001+02:00

After spending too much time re-writing the "Grinding" funcionality - what detects data structures that are common to multiple files or buffers - I´ve released a tarball of version 0.0.4 of the code.

This version of FRET is much improved, I think its the first version that a developer may actually find useful. All junk output has been eliminated and the scanGrind function is now quite smart - it will only detect common structures that are not part of a larger "common structure" (or Gram) and also it will not detect common structures that consist of fill bytes.

The rewrite of the scanGrind function gave me a lot more problems than I thought it would. Eliminating superflous data i.e. sub-Grams of larger Grams and Grams that contain fill bytes was hard to do cleanly. Even now I´m not happy that the implementation is efficient but thats not a concern - fast is not a goal - correct and exhaustive is the goal. The next round of development will probably focus on adding either x86 instruction detection or improved text detection.

Olympus C-770 with GNU/Linux

2005-07-12T14:33:00.001+02:00

Currently working on a HOWTO for using the latest MPEG-4 recording digital cameras with GNU/Linux systems. I have a pretty good perl script that I use to go along with the article but it´ll be a week or two before thats finished. If you use an Olympus C-770 with a GNU/Linux system and would like a script to batch convert your mp4 movies to more usable MPEG-2, AVI or DVD-VOB formats, let me know.

FRET version 0.0.3 Released!

2005-07-08T22:53:00.001+02:00

I have finally published version 0.0.3 on Sourceforge.net. This is definitely the most significant release to date as it now actually provides some useful functionality. Ok, its still pretty limited but its coming along. So if your a looking for a *NIX commandline tool to help you understand the inner workings of an undocumented file format, you now have to look no further.

In this release I have added the hashing of raw data to create a fingerprint of each proposed data structure. This allows for the efficient comparison of structures (Grams). For the hash algorithm I used Landon Curt Noll's FNV algorithm, which he has kindly released in the Public Domain, see Landon's website. This release has also removed the annoying over-zealous detection of offsets within the data. However, there is still lots of work to be done.......

FRET version 0.0.2 Released

2005-06-29T18:55:00.001+02:00

Version 0.0.2 of FRET has gone live on Sourceforge about a half hour ago. I have included some obvious improvements to the commandline tool so the output is now formatted much better. On the libfret side of things, the offset scanning code has been updated substantially, but it is now over-detecting offsets within files - I have to throttle that back in next release. So now its on to working on the next release and firstly what will be included in this - so much to do and so little time. So what is FRET and where is it going?

FRET is a Unix commandline tool that is designed to slot into the Unix tool chain. As there is no point in developing a solution if no hard problem exists, I've looked at a lot of needs within the free software community before going down this road. The problem FRET attempts to solve is that if a developer needs to parse a file format and that format isn't documented (there are lots of potential reasons for this) then (s)he must spend days, weeks or months studying raw files to identify the layout of data within those files. The tools normally utilised to do this are hex editors and binary diff tools. This is inefficient - why not get a program to do what its good at - perform a repetitive, rigerous, boring analysis of a large group of these files and take a guess at the file layout. So thats the end goal: develop a tool that will crunch a large group of files and at the end provide a guess at the layout.

FRET version 0.0.1 Released

2005-06-15T21:30:00.003+02:00

Today I published the first version of FRET. I´ve placed a tarball of the source on the project website. It works ok and does analyse files as planned but is very light on functionality. But you have to start someplace and "release early, release often" is the FOSS motto.

Sourceforge project site up and running

2005-06-14T19:09:00.002+02:00

Sourceforge.net has accepted the proposal for FRET and an initial version of the project website is now in place. Putting the website itself together consisted of....

a two day crash course in xhtml and css stylesheets.
finding a good css template to use for the website, thanks to snop for this.
configuring the sourceforge site which took a bit of time but is all well documented.
implementing the site. all coded by hand using Bluefish. I now know I have to automate most of this process or investigate a CMS.

The site works and is LoFi - just the way I like it.

FRET - helping understand file formats

2005-06-10T15:28:00.001+02:00

fret is a developers tool that analyses multiple files and attempts to identify the data structures within these files. libfret provides the core functionality and is designed to be integrated with other tools. I am currently working on the development of this tool with the aim of publishing the first version within the next 3-4 days. There is a small code clean-up to be done prior to freezing the code for release. Then its package and test and publish.

First Rev design done

2005-05-05T17:20:00.002+02:00

Have completed a first paper design of FRET. It will analyse a group of files in three phases, each phase performing a different task;

Phase 1: Parse each file individually and store detected structures in a database.
Phase 2: Parse each file, comparing its raw data to the already detected structures and use this to identify new structures.
Phase 3: Compare all the detected structures for each file and all the files raw data against each other, identifying new structures.

I´ve also developed (borrowed really) the following terminology to describe whats happening: GRAM - a data structure or pattern of bytes detected in a file or buffer. Each GRAM will have a position, length, type, confidence level and parent. The term GRAM is taken from the classical Greek for a letter. I´m using it because it is the root of the words bigram and trigram that are used in cryptography when performing statistical analysis. FRET, after all, was inspired by coincidence counting - why not treat an unknown file format like an encrypted file for analysis purposes?

FRET - it has a name

2005-05-04T10:23:00.002+02:00

I have seen it and it has a name! Actually, I haven´t seen it but I have mapped it out roughly on paper. Since ealy March, I´ve been investigating needs within the free software community with the idea in mind of starting development of a Unix tool. I've looked at tools such as hex editors and binary diff tools and think I''ve spotted a gap. There is no tool available that will use sheer computing muscle to analyse black-box files and describe their internal structure - this could potentially save man-months if not man-years of development time - now I need to decide if there is a real need.

and then there was 6

2005-05-01T23:32:00.001+02:00

FRET now has 6 phases of analysis. I´ve done some pretty deep thinking (and scribbling) about this and identified 6 phases of analysis and optimised the order of execution. The phases are;

Buffer preprocessing - remove obfuscation etc.
Single Buffer Scans - analyse raw data in each buffer
Raw data to Gram comparison for each buffer
Comparison of Grams for each buffer
Comparison of raw data of each buffer
Rationalise and amalgamate results

Don´t worry, it may all make sense in the end. I´m not sure if this will work either.

KDE Bug 97831

2005-02-19T16:34:00.001+01:00

I've spent the last two weeks burrowing through the internals of KDE, the GNU/Linux desktop in an attempt to see from the inside how KDE development works. I´ve picked Bug 98930 to be my vehicle - a nasty crash during the shutdown of Kivio. Tracking down a bug is the best way to explore unfamiliar code and this bug took a lot a tracing. I´ve finally traced the cause to a race condition between KConfig XT and KLibFactory. I´ve verified the cause and posted a full analysis and demo patch - it will be interesting to see what the maintainer does with this information.