Difference between revisions of "ZIM file format"

The '''ZIM file format''' is based on the old and deprecated [[Zeno File Format]].See also a walk through an example at [[ZIM File Example]].

The '''ZIM file format''' is based on the old and deprecated [[Zeno File Format]].See also a walk through an example at [[ZIM File Example]].

It starts with a header, which is described here:

It starts with a header, which is described here:

== Header ==

== Header ==

Length in byte, all types are littlendian.

Length in byte, all types are littlendian.

{|{{Prettytable}}

{|{{Prettytable}}

| checksumPos || integer || 72 || 8 || pointer to the md5checksum of this file without the checksum itself. This points always 16 bytes before the end of the file.

| checksumPos || integer || 72 || 8 || pointer to the md5checksum of this file without the checksum itself. This points always 16 bytes before the end of the file.

|}

|}

== MIME Type List (mimeListPos) ==

== MIME Type List (mimeListPos) ==

The MIME types in this list are zero terminated strings. An empty string marks the end of the MIME type list.

The MIME types in this list are zero terminated strings. An empty string marks the end of the MIME type list.

{|{{Prettytable}}

{|{{Prettytable}}

| <last entry / end> || string || n/a ||zero terminated|| empty string - end of MIME type list         

| <last entry / end> || string || n/a ||zero terminated|| empty string - end of MIME type list         

|}

|}

== URL Pointer List (urlPtrPos) ==

== URL Pointer List (urlPtrPos) ==

Since directory entries have variable sizes this is needed for random access.

Since directory entries have variable sizes this is needed for random access.

{|{{Prettytable}}

{|{{Prettytable}}

Zimlib caches directory entries and references the cached entries via the URL pointers.

Zimlib caches directory entries and references the cached entries via the URL pointers.

== Title Pointer List (titlePtrPos) ==

== Title Pointer List (titlePtrPos) ==

in the URL pointer list. Note that the title pointers are only 4 bytes. They are not offsets in the file but article numbers.

in the URL pointer list. Note that the title pointers are only 4 bytes. They are not offsets in the file but article numbers.

To get the offset of an article from the title pointer list, you have to look it up in the URL pointer list.

To get the offset of an article from the title pointer list, you have to look it up in the URL pointer list.

{|{{Prettytable}}

{|{{Prettytable}}

* the pointer list is only half in size as 4 bytes are enough for each entry

* the pointer list is only half in size as 4 bytes are enough for each entry

* accessing directory entries by title also makes use of cached directory entries which are referenced by the URL pointers, as implemented in zimlib.

* accessing directory entries by title also makes use of cached directory entries which are referenced by the URL pointers, as implemented in zimlib.

== Directory Entries ==

== Directory Entries ==

There are two types of directory entries: article entries and redirect entries. If the first two bytes are 0xffff the directory entry is a redirect.

There are two types of directory entries: article entries and redirect entries. If the first two bytes are 0xffff the directory entry is a redirect.

=== Article Entry ===

=== Article Entry ===

| parameter || data || ||see parameter len|| (not used) extra parameters                         

| parameter || data || ||see parameter len|| (not used) extra parameters                         

|}

|}

=== Redirect Entry ===

=== Redirect Entry ===

| parameter || data || ||see parameter len|| (not used) extra parameters                         

| parameter || data || ||see parameter len|| (not used) extra parameters                         

|}

|}

== Cluster Pointer List (clusterPtrPos) ==

== Cluster Pointer List (clusterPtrPos) ==

| ... || integer || ... || 8 || ...                           

| ... || integer || ... || 8 || ...                           

|}

|}

== Clusters ==

== Clusters ==

To find the data of a specific directory entry within a cluster the uncompressed cluster has a list of pointers to blobs within the uncompressed cluster after the first byte.

To find the data of a specific directory entry within a cluster the uncompressed cluster has a list of pointers to blobs within the uncompressed cluster after the first byte.

{|{{Prettytable}}

{|{{Prettytable}}

The offset addresses uncompressed data. The last pointer points to the end of the data area. So there is always one more offset than blobs. Since the first offset points to the start of the first data, the number of offsets can be determined by dividing this offset by 4. The size of one blob is calculated by the difference of two consecutive offsets.

The offset addresses uncompressed data. The last pointer points to the end of the data area. So there is always one more offset than blobs. Since the first offset points to the start of the first data, the number of offsets can be determined by dividing this offset by 4. The size of one blob is calculated by the difference of two consecutive offsets.

== Namespaces ==

== Namespaces ==

They can be distinguished by prepending the article namespace before the article name in the URL path, eg. ''http://localhost/A/Articlename''.

They can be distinguished by prepending the article namespace before the article name in the URL path, eg. ''http://localhost/A/Articlename''.

{|{{Prettytable}}

{|{{Prettytable}}

| X || fulltext index - see [[ZIM Index Format]]         

| X || fulltext index - see [[ZIM Index Format]]         

|}

|}

== URLs ==

== URLs ==

The URLs in the UrlPointerlist are not encoded. Some readers process the requests that already do the decoding internally whereas most readers will handle the URLs directly. In this case you have to do the decoding before you pass the parameter to zimlib, but zimlib already provides a method to do so.

The URLs in the UrlPointerlist are not encoded. Some readers process the requests that already do the decoding internally whereas most readers will handle the URLs directly. In this case you have to do the decoding before you pass the parameter to zimlib, but zimlib already provides a method to do so.

=== Local Anchors ===

=== Local Anchors ===

Should you render the article contents by yourself you have to consider this and take care of it before you hand requests to zimlib.

Should you render the article contents by yourself you have to consider this and take care of it before you hand requests to zimlib.

== Encodings ==

== Encodings ==

Old Zeno files used a mixture of Latin1 and UTF-8 so there is still some "auto detection" code left in the ''zimlib'', a workaround for this bug. This will be removed in future versions. Zeno files are not supported anymore.

Old Zeno files used a mixture of Latin1 and UTF-8 so there is still some "auto detection" code left in the ''zimlib'', a workaround for this bug. This will be removed in future versions. Zeno files are not supported anymore.

=== Integer Encoding ===

=== Integer Encoding ===

Old Zeno files used the QUnicode library instead. By switching to UTF-8 the new format is more standard-adherent and easier to understand.

Old Zeno files used the QUnicode library instead. By switching to UTF-8 the new format is more standard-adherent and easier to understand.

== See also ==

== See also ==