Archive

Older writings that are fading into history......

FRET version 0.0.4 released!

[Fri, 22 Jul 2005] After spending too much time re-writing the "Grinding" funcionality - what detects data structures that are common to multiple files or buffers - I´ve released a tarball of version 0.0.4 of the code.

This version of FRET is much improved, I think its the first version that a developer may actually find useful. All junk output has been eliminated and the scanGrind function is now quite smart - it will only detect common structures that are not part of a larger "common structure" (or Gram) and also it will not detect common structures that consist of fill bytes.

The rewrite of the scanGrind function gave me a lot more problems than I thought it would. Eliminating superflous data i.e. sub-Grams of larger Grams and Grams that contain fill bytes was hard to do cleanly. Even now I´m not happy that the implementation is efficient but thats not a concern - fast is not a goal - correct and exhaustive is the goal. The next round of development will probably focus on adding either x86 instruction detection or improved text detection.

Olympus C-770 with GNU/Linux

[Tue, 12 Jul 2005] Currently working on a HOWTO for using the latest MPEG-4 recording digital cameras with GNU/Linux systems. I have a pretty good perl script that I use to go along with the article but it´ll be a week or two before thats finished. If you use an Olympus C-770 with a GNU/Linux system and would like a script to batch convert your mp4 movies to more usable MPEG-2, AVI or DVD-VOB formats, let me know.

FRET version 0.0.3 Released!

[Fri, 08 Jul 2005] I have finally published version 0.0.3 on Sourceforge.net. This is definitely the most significant release to date as it now actually provides some useful functionality. Ok, its still pretty limited but its coming along. So if your a looking for a *NIX commandline tool to help you understand the inner workings of an undocumented file format, you now have to look no further.

In this release I have added the hashing of raw data to create a fingerprint of each proposed data structure. This allows for the efficient comparison of structures (Grams). For the hash algorithm I used Landon Curt Noll's FNV algorithm, which he has kindly released in the Public Domain, see Landon's website. This release has also removed the annoying over-zealous detection of offsets within the data. However, there is still lots of work to be done.......

FRET version 0.0.2 Released

[Wed, 29 Jun 2005] Version 0.0.2 of FRET has gone live on Sourceforge about a half hour ago. I have included some obvious improvements to the commandline tool so the output is now formatted much better. On the libfret side of things, the offset scanning code has been updated substantially, but it is now over-detecting offsets within files - I have to throttle that back in next release. So now its on to working on the next release and firstly what will be included in this - so much to do and so little time. So what is FRET and where is it going?

FRET is a Unix commandline tool that is designed to slot into the Unix tool chain. As there is no point in developing a solution if no hard problem exists, I've looked at a lot of needs within the free software community before going down this road. The problem FRET attempts to solve is that if a developer needs to parse a file format and that format isn't documented (there are lots of potential reasons for this) then (s)he must spend days, weeks or months studying raw files to identify the layout of data within those files. The tools normally utilised to do this are hex editors and binary diff tools. This is inefficient - why not get a program to do what its good at - perform a repetitive, rigerous, boring analysis of a large group of these files and take a guess at the file layout. So thats the end goal: develop a tool that will crunch a large group of files and at the end provide a guess at the layout.

FRET version 0.0.1 Released

[Wed, 15 Jun 2005] Today I published the first version of FRET. I´ve placed a tarball of the source on the project website. It works ok and does analyse files as planned but is very light on functionality. But you have to start someplace and "release early, release often" is the FOSS motto.

Sourceforge project site up and running

[Tue, 14 Jun 2005] Sourceforge.net has accepted the proposal for FRET and an initial version of the project website is now in place. Putting the website itself together consisted of....

a two day crash course in xhtml and css stylesheets.
finding a good css template to use for the website, thanks to snop for this.
configuring the sourceforge site which took a bit of time but is all well documented.
implementing the site. all coded by hand using Bluefish. I now know I have to automate most of this process or investigate a CMS.

The site works and is LoFi - just the way I like it.

FRET - helping understand file formats

[Fri, 10 Jun 2005] fret is a developers tool that analyses multiple files and attempts to identify the data structures within these files. libfret provides the core functionality and is designed to be integrated with other tools. I am currently working on the development of this tool with the aim of publishing the first version within the next 3-4 days. There is a small code clean-up to be done prior to freezing the code for release. Then its package and test and publish.

and then there was 6

[Wed, 01 Jun 2005] FRET now has 6 phases of analysis. I´ve done some pretty deep thinking (and scribbling) about this and identified 6 phases of analysis and optimised the order of execution. The phases are;

Buffer preprocessing - remove obfuscation etc.
Single Buffer Scans - analyse raw data in each buffer
Raw data to Gram comparison for each buffer
Comparison of Grams for each buffer
Comparison of raw data of each buffer
Rationalise and amalgamate results

Don´t worry, it may all make sense in the end. I´m not sure if this will work either.

First Rev design done

[Wed, 05 May 2005] Have completed a first paper design of FRET. It will analyse a group of files in three phases, each phase performing a different task;

Phase 1: Parse each file individually and store detected structures in a database.
Phase 2: Parse each file, comparing its raw data to the already detected structures and use this to identify new structures.
Phase 3: Compare all the detected structures for each file and all the files raw data against each other, identifying new structures.

I´ve also developed (borrowed really) the following terminology to describe whats happening: GRAM - a data structure or pattern of bytes detected in a file or buffer. Each GRAM will have a position, length, type, confidence level and parent. The term GRAM is taken from the classical Greek for a letter. I´m using it because it is the root of the words bigram and trigram that are used in cryptography when performing statistical analysis. FRET, after all, was inspired by coincidence counting - why not treat an unknown file format like an encrypted file for analysis purposes?

FRET - it has a name

[Wed, 04 May 2005] I have seen it and it has a name! Actually, I haven´t seen it but I have mapped it out roughly on paper. Since ealy March, I´ve been investigating needs within the free software community with the idea in mind of starting development of a Unix tool. I've looked at tools such as hex editors and binary diff tools and think I''ve spotted a gap. There is no tool available that will use sheer computing muscle to analyse black-box files and describe their internal structure - this could potentially save man-months if not man-years of development time - now I need to decide if there is a real need.

KDE Bug 97831

[Thu, 23 Feb 2005] Before working on bug 98930, this was the first KDE bug I worked on - a crash during the shutdown of KDevelop. I now believe that the cause is the same as 98930 - an inherint design flaw in KStaticDeleter - it assumes deleteStaticDeleters is only every called once. From a quick peruse of the bug database I believe that several KDE applications that crash during shutdown suffer from this same flaw. I also believe from some posts I have seen that due to egos/time-constraints this root cause will take some time to be eliminated from KDE.

Up to my ears in KDE internals

[Tues, 19 Feb 2005] I've spent the last two weeks burrowing through the internals of KDE, the GNU/Linux desktop in an attempt to see from the inside how KDE development works. I´ve picked Bug 98930 to be my vehicle - a nasty crash during the shutdown of Kivio. Tracking down a bug is the best way to explore unfamiliar code and this bug took a lot a tracing. I´ve finally traced the cause to a race condition between KConfig XT and KLibFactory. I´ve verified the cause and posted a full analysis and demo patch - it will be interesting to see what the maintainer does with this information.

mick mccarthy / software engineer

.....archive.....

Links: