JigdoFUSE

Description

JigdoFUSE uses FUSE to provide images generated from Jigdo files.

Motivation

Images of Debian (or any major Linux distibution) consume lots of disk space and they consist mainly of packages usually hosted on the same server. The aim is to provide the way to host images at lower disk space cost. (In fact I wrote it to save my disk space. ;) And to share more Debian images (even incomplete) through BitTorrent.)

Jigdo or Jigsaw Download is a tool for creating a large file from its parts. It takes two files as input: a jigdo file which maps parts' MD5 sums with URL and a template which describes the large file itself. Jigdo can construct CD images given these two files and a set of parts, which are usually files form the image. Some images may share parts, so it is more space efficient to keep parts only, and generate images on-the-fly.

FUSE or Filesystem in Userspace allows creation of virtual filesystem which is created by an userspace program. This means one can put anything into files in that filesystem. In this case it's the result of Jigdo-like program which generates images given the same input as Jigdo.

Instructions

Suppose we want JigdoFUSE to offer an image from debian.jigdo. Firstly we need all parts the image is made of or the image itself. We must arrange the parts so JigdoFUSE can see them. This can by done with a helper script jigdo-files.sh. If we don't have all the files nor the image I recommend using jigdo-lite from jigdo-file package to get them.

Now let's make a configuration file. Syntax of an entry is simple:
[three-letter type][delimiter][value]
The type is one of:

Delimiter is currently any character (may change in future). Value depends on an entry type:

Example configuration file:
LAB:Debian=/home/old/pkg/Debian\:
/LAB:Non-US=/home/old/pkg/Non-US\:
JIG:/home/old/jigdo-cd/debian-testing-amd64-CD-1.jigdo
JIG:/home/old/jigdo-cd/debian-testing-i386-CD-1.jigdo
LAB:ora=/to/ry

This defines two jigdo files which share a single label.
First line is a valid label definition. Note the escaped colon in [base path] part and another colon as a separator, which is not escaped.
Second line will be rejected with warning, as '/LA' is not 'LAB' nor 'JIG'.
Lines 3 and 4 define jigdo files to be offered.
Last line defines a valid label, but it will not be applied to JIG entries above, so it is not functional. Was there a JIG entry below, it would use both valid labels.
If a LAB entry with previously used label is encountered, the labels binding is overriden. Changing back to the same binding is supported without memory penalty, though it can be avoided in most cases.

Now it is time to mount the filesystem. Syntax is:
JigdoFUSE $conffile $mountpoint [-o $options]
For example:
JigdoFUSE jdf.conf /mnt/jdf -o ro,allow_other
JigdoFUSE filesystem defined in jdf.conf would be mounted in /mnt/jdf and available for all users to read. FUSE requires root privileges and a fuse kernel module to mount. We can also create an fstab entry:
JigdoFUSE#/etc/jdf.conf /mnt/jdf fuse ro,allow_other 0 0

Preformance

For now the performance is poor - reading a CD image takes 4 to 6 minutes on my laptop. It is going to be improved in the future.

Future

Files

namerequirementsdescription
main.c currently: libfuse, libbz2, zcat. Source of JigdoFUSE is currently this single ~1000-line file. Compile with $(gcc main.c `pkg-config fuse --cflags --libs` -lbz2)
jigdo-files.sh See: [utils section] This script helps to create a partial mirror with packages required by a jigdo file. See: jigdo-files.sh man
template file specification [obsolete - the real specification exists]
This is my interpretation on what a jigdo template file is. It may be inaccurate or incomplete.
main9IV2010.c libfuse, libbz2. Vast mount speed improvement thanks to Dr. Richard Atterer (inventor of Jigsaw Download).
Compile with $(gcc main.c `pkg-config fuse --cflags --libs` -lbz2)
A little anniversary: one round month of development! :D
JigdoFUSE_090410_amd64 fuse Linux module, zcat. Binary version for amd64 with debug symbols. i386 builds are unavailable until I learn how to cross compile.
main11IV2010.c libfuse, libbz2, zlib. DATA entries support added. Now more images work eg. debian-update_5.0.1 and some older ones.
Compile with $(gcc main.c `pkg-config fuse --cflags --libs` -lbz2 -lz)
JigdoFUSE_110410_amd64 fuse Linux module, zcat. Binary version for amd64. i386 builds are unavailable until I learn how to cross compile.
Although compiled with zlib it still needs zcat to decompress jigdos. I'll try to eliminate this problem in the next release.
main12IV2010.c libfuse, libbz2, zlib. Template format version 1.0 support added, but not tested yet (couldn't find any files in that format).
Compile with $(gcc main.c `pkg-config fuse --cflags --libs` -lbz2 -lz)
JigdoFUSE_120410_amd64 fuse Linux module, zcat. Binary version for amd64. i386 builds are unavailable until I learn how to cross compile.
Still with zcat dependency.
jigdofuse-0.5.tar.gz libfuse >= 2.6, libbz2, zlib; fuse Linux module, zcat. My first tarball. The program is still unstable. It sometimes hangs at mount. I did not notice any crashes though.