jigdo template

This document describes a jigdo template. Examples are based on Debian testing image retrieved 16II2010.

Jigdo template is a file which describes the structure of an image and contains data that is not filled by parts.

Specification

This may be incomplete, inaccurate or misleading and is by no means official. In fact this page was writen because author of JigdoFUSE was too shy to ask for the real specification.

Jigdo template file consists of:

  1. Comment.
  2. One or more BZIP entries.
  3. DESC section.

in that order.

Comment

Comment is a string of printable characters ended with double Windows newline (0x0d0a0d0a)
example:

00000000  4a 69 67 73 61 77 44 6f  77 6e 6c 6f 61 64 20 74  |JigsawDownload t|
00000010  65 6d 70 6c 61 74 65 20  31 2e 31 20 4a 54 45 2f  |emplate 1.1 JTE/|
00000020  31 2e 31 38 20 0d 0a 4a  54 45 20 61 74 20 68 74  |1.18 ..JTE at ht|
00000030  74 70 3a 2f 2f 77 77 77  2e 65 69 6e 76 61 6c 2e  |tp://www.einval.|
00000040  63 6f 6d 2f 7e 73 74 65  76 65 2f 73 6f 66 74 77  |com/~steve/softw|
00000050  61 72 65 2f 4a 54 45 2f  20 3b 20 6a 69 67 64 6f  |are/JTE/ ; jigdo|
00000060  20 61 74 20 68 74 74 70  3a 2f 2f 61 74 74 65 72  | at http://atter|
00000070  65 72 2e 6e 65 74 2f 6a  69 67 64 6f 2f 20 0d 0a  |er.net/jigdo/ ..|
00000080  0d 0a                                             |..              |
00000082
	 

Newlines are allowed inside the comment as long as they are not followed by another newline - this would truncate the description and corrupt the interpretation of the rest of template file.

BZIP entries

A single BZIP entry is constructed as follows:

offset (bytes)size (bytes)meaning
0x00 4 "BZIP" type string. A BZIP entry always begins with it.
0x04 6 Entry length. This is the length of whole entry, type string inclusive. It can be used to compute an offset of the next BZIP entry as well as the size of compressed data.
0x0a 6 Data length. Size of uncompressed data.
0x10 entry_length - sum(sezes_of_the_rest_of_fields)
or:
entry_length - entry_length_offset
Data. Bzip2 compressed data itself.

All BZIP entries' data uncompressed and concatenated constitute a template data.
example:

0x000082  42 5a 49 50|22 e5 00 00  00 00|00 10 0e 00 00 00  |BZIP"...........|
	 

DESC section

This part of template file is a key to produce a valid image. It has a header, many entries and a footer.

header
offset (bytes)size (bytes)meaning
0x00 4 "DESC" type string.
0x04 6 Desription section length. Length of all parts included.
entry
offset (bytes)size (bytes)meaning
0x00 1 Type byte.
valuemeaning
0x02 in-template
The entry tells how many bytes of data contained in template data should be put into an image.
0x06 need-file
The entry describes what file to put the data from.
0x01 6 data length. Size of data to be put into image. For need-file entries it is usually length of file needed.
0x07 8 [need-file only] rsync sum of a file
0x0f 16 [need-file only] MD5 sum of a file
footer
offset (bytes)size (bytes)meaning
0x00 1 type byte. The footer is basically an entry, and thus has similar construction. Here type byte has a value of 0x05 - image-info.
0x01 6 Image length.
0x07 16 Image MD5 sum.
0x17 4 block size. I don't know the purpose of this
0x1b 6 Desription section length. Length of all parts included. It's the same value as in header. It's probably for integrity check.


example:

0x84d9be	44 45 53 43  AC A8  00 00 00 00
0x84d9c8	02  00 B0 DA 00 00 00  |DESC......|
0x84d9ef	06  48 BA 40 00 00 00  6A 7F BF F8 72 7C E9 94
0x84d9de	F1 0A F8 54 B7 58 06 45 90 BF 83 CC 36 B4 7E 45
0x84d9ee	02  B8 05 00 00 00 00 
0x84d9f5	06  4C DA 26 00 00 00  A4 10 63 B2 D6 E3 73 BA
0x84da04	AA C2 9E 07 2A BE DE 73  5F 1B 0B BC DC 15 EE 32 
...
0x858249	05  00 38 70 28 00 00
0x858250	B7 D0 92 9A BA 74 2D 83 23 18 17 E0 07 6D 13 FC)
0x858260	00 04 00 00  AC A8 00 00 00 00
	

How to get data

The template file describes the overall outline of the image as well as fills the gaps between files. It is, however, a bit complicated to extract data from it.

To illustrate the process we will try to extract data of given size at 0x011b6abc. Similar algorithm is now implemented in JigdoFUSE.

First we need to seek the DESC section. Then we search for an entry that matches our needs. If the offset was 0 we wouldn't have to skip any template data (skip ← 0). For each entry we compare its length with a given offset. If offset is not smaller than length we can skip the entry and proceed, but first we need to decrement offset by length (offset ← offset - length), and - if entry was of in-template type - remember that we will need to skip length bytes more of template data (skip ← skip + length). When we reach the point in which offset is smaller than length, the current entry describes data we want to get.

If the entry is of need-file type, the data we seek is at offset (remaining after subsequent decrementations) of the file with MD5 sum supplied by current entry. Now we must refer to the jigdo file to check where the file is.

It is more complicated when the entry is of in-template type. Now we must search for BZIP entry which corresponds to template data at skip + offset. The procedure is similar to the previous one, but now entries are of various size.

Let the new offset in template date be the sum of current offset (after decrementations) and amount of template data to skip (skip ← offset + skip) [sic! skip was initially an offset in template data, so I'll leace it like this]. For each BZIP entry we compare the length of uncompressed data with template data offset. If length is smaller, decrement the Template data offset by length (skip ← skip - length) and proceed to the next BZIP entry (use compressed data length to calculate where it begins). Otherwise decompress the data and read the part that is at skip (remaining after decrementations).

I both cases we can get only length - offset bytes of data. Then we must proceed to the next DESC entry and repeat the procedure for 0 offset and for template data offset and size both smaller by the ammount of data already read (size ← size - length + offset).