KaZaA P2P FastTrack File Formats - 2003-10-12

Experiment with the KazaaNet v1.1 Supernode Hopping Tool. Try to make your KaZaA connect to other Continents.

Active Downloads (.DAT), Local Meta Database (.DBB), SuperNode List

Keywords: KazaaNet, P2P, CDE, Joltid, KaZaA, FastTrack, Internals, Protocol, Search Cache, Grokster, iMesh fileshare, .DAT, .DBB, Grid, Distributed, FileSearch, SuperNode, Hash
See Also: Content Distribution Environment by Joltid FastTrack Network Control
See Also: Microsoft Peer-to-Peer Networking, in an IPv6 Advanced Networking Pack for Windows 5.1sp1
See Also: giFT-FastTrack plugin, open source by Markus Kern et all
See Also: Grid.org, the largest public grid powered by United Devices Grid MP Global
See Also: Structure of .dat files, DATview by idleloop@yahoo.com
See Also: The Globe Project, distributed shared objects, by Vrije Universiteit
See Also: MorpheusX Dat Repairer, includes C++ source
See Also: www.distributed.net, mathematics number crunching, uses encryption and authentication ciphers, started in 1997
See Also: Pastry, a substrate for peer-to-peer applications, by Rice University and Microsoft Research
See Also: Sig2Dat by vlaibb@yahoo.com
See Also: KaZaA Lite Forum by my-k-lite
See Also: alt.internet.p2p by your newsfeeder, or via google.com
See Also: alt.music.mp3.kazaa by your newsfeeder, or via google.com
See Also: Global Grid Forum
See Also: Peer-to-peer development by p2p-hackers Mail List server
See Also: Zeropaid.com forums
See Also: Altnet Resource Network by Brilliant Digital Inc.


KaZaA 2.[0,1] FastTrack v3, Whadde Hek...

A. FastTrack download*.DAT Appendix Format

B. FastTrack data*.DBB (Local Shared Objects Meta Database) File Format

C. Some known FastTrack Kazaa Meta Tags

D. KazaaNet Local SuperNode List Cache

Kazaa FastTrack files are organised as Sequences of Items. Items may be simple types or subsequences like Lists, tables, strings. Some items have a fixed length. Some items are prefixed with a count. Some items are terminated by a special constant value (asciiz/ansiz strings, chunklist).

All Numbers are serialised in Intel little endian byte order (least significant byte first). All Texts are strings of single byte characters (from the current user's MS Windows 'ANSI' codepage).

A. KaZaA FastTrack download*.DAT Appendix Format

A .DAT file is a staging file with a filename following a locally unique pattern like 'download103988081426526625.dat', that is produced by the FastTrack Download Manager in the client application's download directory. It extends in size as more data retrieval ranges at the end of the file are assigned to one or more Download Sources. Received data chunks (from the responses sent by peer nodes on HTTP GET range requests) are written directly into the .DAT file at the corresponding file offset. While the download is busy, a FastTrack Appendix with download job state information is present at the end of the file and contineously maintained. When all chunks of the file are received and written completely, that Appendix is removed and the .DAT file is renamed to the final filename that was selected for download in the application's search list.

Table 1: Format of FastTrack .DAT Appendix sequence:

# Name Type Values Description
1 datState DWORD 0 (=Active), 1 (=Paused), 2 (=MoreSourcesNeeded?) Global state of this file download job.
2 datSourceCount DWORD 1... Count of composite Source subsequences in the list included hereafter.
3

datSourceList: 1 or more concatenated variable length Source subsequences, each with this layout:

1 srcRemoteFileName ANSIZ "Tiken Jah Fakoli - Y'en A Marre.mp3", '\0' Variable length name (no path) of the file at the remote source node. Can be empty ('\0' only).
2 srcDownloadURL ANSIZ '\0' Variable length filepath. Always empty in Kazaa 2.[0,1] ('\0' only).
3 srcFileID DWORD 3731 Small unique number, typically identical for a given file contents at all Sources. Used as virtual subdirectory name in HTTP GET requests like "http://<KazaaHost>:<KazaaPort>/3731/Tiken%20Jah%20Fakoli....mp3". Such URL's were used for file exchanges with peer nodes in Kazaa 1.x (only ?). Kazaa 2.[0,1] still returns a list of these URL's in a HTML page table produced by a local http://localhost:<KazaaPort>/ GET root request.

4 srcContentHash BYTE[20] 16-byte crypto std. MD5 one-way digest + 4-byte FastTrack smallhash DWORD Fingerprint for file content. Files with identical content have an identical Message Digest value. No different content anywhere should have this same cryptographic secure, one-way MD5-value. FastTrack smallhash adds more content identifying bits. Not all file data is included in the hash calculation.
5 srcFileSize DWORD 3751893 Size of the remote file at this Source. Counted in bytes.
6 srcNodeInetNum DWORD 0x04,0x03,0x02,0x01 = 0x01020304 = "1.2.3.4" 32-bit IP address number of this FastTrack peer Source node.
7 srcNodeTcpPort DWORD 0xF1,0x0A,0x00,0x00 = 0x0af1 = 2801 TCP file service port of FastTrack peer file Source. Default KaZaA file service IP-port is 1214. Can be NULL for a file service behind a proxy server and/or firewall that does not allow the remote source peer to accept incoming TCP connections. In these cases the destination node sends a PUSH Request to his supernode. This PUSH request is mediated through the (network of) supernode(s) to the source node. The source node then initiates a new TCP connection to the destination peer. Over this connection, the source sends a GIVE nnnn command to tell the destination that it can start the actual HTTP GET file download session on this same connection.
8 srcSuperNodeInetNum DWORD 0x04,0x03,0x02,0x01 = 0x01020304 = "1.2.3.4" 32-bit IP address number of the FastTrack SuperNode to which this peer Source node is connected.
9 srcSuperNodePort DWORD 0x32,0x09,0x00,0x00 = 0x0932 = 2354 TCP or UDP supernode service port of FastTrack SuperNode to which the peer file Source is connected.
10 srcName BYTE[69] "p2pWillNever Die@KaZaA",'\0' Username and Networkname, concatenated with an '@'-character and terminated by a NULL char. Variable length ASCIIZ string, written in a fixed length block of 69 bytes. After the string terminator, only insignificant bytes are present to fill up the space. These filler bytes are random leftovers from the application stack or heap and should be ignored. Networknames can be KaZaA, Grokster, fileshare, or whoever licenses the FastTrack Protocol (and C++ lib) stack (Joltid 2nd generation Peer to Peer technology). The latter is for iMesh. Old Morpheus used MusicCity.
11 srcKbps DWORD 122 Transfer speed in Kilo bits per second. Initial value for this source property is received from the Supernode network. Updated ever after all response data of an actual TCP HTTP GET <range> request is received and periodically in between.
12 srcKbpsLastModified DWORD 0x3e1354a1, time in seconds elapsed since 1970-01-01T00:00:00Z UTC (C/Unix Epoch time_t) The datetime at which the previous field (srcKpbs) was last updated. Can be NULL, to signal no transfer speed was calculated (yet).
13 srcUnknown1 DWORD 0 Seems to be NULL always.
14 srcGroup DWORD 0, 1 or 2 Probably the Source Group to which this Source belongs. Source Groups seem to be created when different Sources do not provide the same file content: srcFileID, srcContentHash and/or srcFileSize differ.
15 srcRetry DWORD 0 or 1 or 3 or ... Could be a TCP connect retry counter, or some Download Mode (0=inactive, 1=push, 3=direct) ?
16 srcUnknown2 DWORD 0 Seems to be NULL always.
17 srcUnknown3 BYTE 0 Seems to be NULL always.
18 srcUnknown4 BYTE 1 Seems to be 1 always.

End of Source subsequence.

4 metaTagCount DWORD 1... Count of metaTag items (file content descriptive properties) in the unordered set enlisted hereafter. These FastTrack metaTag sets play a role everywhere. They are used in the actual TCP HTTP GET file transfer, they are included in the File Search Cache maintained by the network of Supernodes, and they are included in the FastTrack .DBB Local Shared Files Meta Database Files (See next section).
5

Each metaTag item is a variable length subsequence, with this layout:

1 metaTagID DWORD See the FastTrack MetaTagList A number that identifies the metaTag's Value format and meaning. Currently (jan 2003) assigned FastTrack tagID numbers seem to range from 1 up to ca. 60. They are mapped on .MP3 tags, .PDF and .DOC file properties, etc. One tag is always present: tagID #3, with the 20-byte FastTrack ContentHash (see also 3.3-srcContentHash in this table).
2 metaTagLength DWORD typically 4 (for DWORD types), or greater (for ANSIZ types) Count of bytes in next metaTagValue.
3 metaTagValue BYTE[tagLength] 128 Content of this metaTag.

End of metaTag item subsequence layout.

6 datStartTime DWORD time_t, seconds since Epoch Appears to always be earlier than the .DAT file's creation time. Could be the datetime of instantiation of the application process that created this .DAT file. Or the datetime (age) of the Search result from which this .DAT file's download job was finally created ?
7 datUnknown1 DWORD[2] 0xFFFFFFFF, 0 Seem to be -1 and 0 always.
8 datLocalFilePath ANSIZ "C:\Kazaa\ Downloads\ Tiken...mp3",'\0' Fully qualified (path included) local filename. This is the name the .DAT file is renamed to when the download is completed.
9 datUnknownTime1 DWORD 0 or time_t Mostly 0. Could be the datetime when something is locked in this .DAT ?
10 datUnknown2 DWORD 1 Seems to be 1 always. Indicates following chunkRange tables are for Source Group # 1 ?
11 datCompletedChunksCount DWORD 1... Multi source, swarming downloads are performed in HTTP GET file byte Ranges. The resulting chunks that are completely received are enlisted hereafter in the chunkRange item format.
12

Each chunkRange item is a fixed length subsequence, with this layout:

1 rangeStart DWORD 0... Chunk offset in source file = chunk offset in .DAT file = HTTP GET byte Range Start
2 rangeSize DWORD 0... Chunk size in source file = chunk size in .DAT file = HTTP GET byte Range Length. Can be NULL, probably meaning that this chunkRange item isn't used (anymore).

End of chunkRange item subsequence layout.

13 datFullRange DWORD[2] 0, <FileSize> This rangeStart + rangeSize always seems to reflect the whole final file to produce.
14 datRangeShift BYTE 0 always Seems to be 0 always.
15 datRangeStates DWORD[2][1...] 0,1096756 Variable number of DWORD pairs. Seem to be rangeStart, rangeSize pairs. Could indicate some postprocesing state of completed ranges (maybe ContentHash precalculation ?). The number of pairs seems to be related with the number of Sources. Must be very significant data, because a .DAT file can grow with 8 bytes just to make room for another completed rangepair and thus also to preserve space for the items in this subsequence here. The last element in this array has a rangeSize value of NULL. A last element is always present (can be {NULL, NULL}).
16 datAppendixSignature DWORD constant value: 'K','A','Z','A' = 0x4B,0x41,0x5A,0x41 = 0x415a414b To recognize a .DAT appendix in a file.
17 datAppendixSize WORD 836 Size of this appendix counted in bytes from its start at field datState up to, but not including, this field's previous field datAppendixSignature. Is the offset to the beginning of this appendix in the .DAT file when positioning the read pointer backwards from (end of the file - 10).
18 datAppendixChecksum DWORD 0x7BAF201F To verify the integrity of the contents of this .DAT appendix. Kind of CRC32 value. The calculation algorithm uses a constant/hardcoded lookup table (table size: DWORD[256] = BYTE[0x400]). Kind of fast/small hash. Is the same algorithm used to calculate a srcFileID and the last 4 bytes of a srcContentHash ?

B. KaZaA FastTrack data*.DBB (Local Shared Objects Meta Database) File Format

The FastTrack application's Local Content Database files contain meta information on the files that are shared/published/uploaded/advertised/contributed by the local machine on the FastTrack P2P network. These index files are maintained by Kazaa's LocalContentManager. An active Kazaa process permanently monitors the local folders that are shared, any File Add, Change and Delete event is reflected in the meta database files.

Meta info items are filename, foldername, filesize, and the filetime. In addition there are a variable number of file tags that describe the contents of the file. FastTrack Kazaa tags include mappings for most .MP3, .DOC, .PDF, .AVI and other popular Media file content properties. This file meta info is used in the FastTrack distributed network File Search protocol. An essential tag is the FastTrack ContentHash. The value represents a 'fingerprint' of the content, worldwide unique for each specific dataset. In Kazaa 2.[0,1] it is the (only) tag used in the Download URL (a HTTP GET Download request like http://peerIP:peerPort/.hash=ab56..83fe).

When a FastTrack client node (re)connects to (a selected and available supernode in) the virtual P2P network, the client node sends his database files to the supernode. This index replication probably isn't incremental. The supernode will make the list of shared files available for searching by other supernodes. This way the meta gets published in the distributed FastTrack file Search Cache.

KaZaA 2.[0,1] will recreate the meta database files at each new start, should they not exist already/anymore in the configured Database Directory. The pathname of that folder is kept in the Windows Registry at HKLM\Software\KaZaA\LocalContent\DatabaseDir.

Executing URL http://localhost:<KazaaPort>/ makes the Kazaa 2.[0,1] application generate a HTML-page with a table of contents showing the entries currently present in the LocalContent Database files [replace <KazaaPort> with the configured TCP port for the Kazaa download service, the default value is 1214]. This URL works with any HTTP Browser that runs on the same machine as the one Kazaa is running on. The Kazaa 2.[0,1] built-in HTTP-server responds with a "404 Not Found" HTTP-error when such a HTTP GET root request originates from a remote machine. Earlier versions of KaZaA reportedly do respond to such remote requests.

.DBB files contain arrays of fixed length slots.

Table 2: Format of FastTrack .DBB Meta Database file slot:

# Name Type Values Description
1 dbbsLabel DWORD constant 'l','3','3','l' Slot signature label.
2 dbbsLength DWORD 367 Count of bytes effectively used in this slot after this field. Can be NULL.
3 dbbsFileName ANSIZ "Tiken ...mp3", '\0' Local name of shared file.
4 dbbsFolderName ANSIZ "C:\Kazaa\Shared Files", '\0' Name of local filesystem directory where this slot's shared file exists.
5 dbbsFileSize DWORD 4012684 Size of file, counted in bytes.
6 dbbsFileTime DWORD time_t, seconds since Epoch (1970-01-01T00:00:00U) UTC/GMT Datetime file was last modified.
7 dbbsShareTime DWORD 0 or time_t ? LastDatetime the file was available for sharing ? Default 0. Upload count ? Popularity ?
8 dbbsUnknown DWORD 0 or 1 Mostly 0. Can be 1 ?
9 dbbsFlag BYTE 1 Seems to be 1 always. Flag ? Indicates sharing state ? Upload count ? Popularity ? Modified since download ?
10 metaTagCount DWORD 1... See metaTagCount layout in Table 1.
11

Each metaTag item is a variable length subsequence, see metaTag item layout in Table 1.

12 dbbsFiller BYTE[] All 0 for fresh (first time used) slots in newly created .DBB files. Slot space fill bytes. Slot space depends on .DBB file: either db256.dbb, db1024.dbb or db2048.dbb.


C. Some known Kazaa FastTrack Meta Tags

Table 3: Some Kazaa FastTrack Meta Tag Values:

ID Name Type Values Description
1 year DWORD 2002 Release year
2        
3 hash BYTE[20] 0x56,0x98,....,0xF3,0xBA FastTrack File Content Hash (16 byte MD5 Digest + 4-byte FTsmallHash)
4 title ANSIZ "This track", '\0' Song track title
5 length DWORD 212 Media play length (duration) in seconds
6 artist, author? ANSIZ "Tiken", '\0' Media file artist, author
7        
8 album ANSIZ "The Works", '\0' Media album, title of track/song collection
9        
10 language ANSIZ "en", '\0' 2-character ISO Language Code "en", "es", "fr", "de", "nl"
11        
12 keywords ANSIZ "fasttrack;formats;meta", '\0' List of keywords.
13 resolution DWORD[2] 1280, 1024 Screen resolution for video files: 2048x1536, 1280x1024, 1024x768, 800x600, 640x480, 320x240, 160x120, 80x60.
14 category ANSIZ "Rhythmic Soul", '\0' Dance Hall, Euro-House, A capella, Drum Solo, ..., Disco, Dance, Country, Classic Rock, Blues
15        
16 os ANSIZ   operating system name
17 colours DWORD 24 ? Screen colour depth in bits ?
18 type ANSIZ "Thesis", '\0' Media type description.
19        
20        
21 quality DWORD 192 Audio sample bitrate:
320 kbps (CD quality), 256 kbps, 224 kbps, 192 kbps, 160 kbps, 128 kbps, 112 kbps,
96 kbps (Normal quality), 80 kbps, 64 kbps, 56 kbps, 48 kbps,
32 kbps (Telephone quality), 24 kbps, 20 kbps.
22        
23        
24 version ANSIZ "5.00", '\0' Software version.
25        
26 comment ANSIZ "Ripped by FastRip", '\0' User comment.
27        
28 codec ANSIZ "Fraunhofer", '\0' Name of media stream coder/decoder.
         
53 integrity DWORD 0=Poor, 1=Average, 2=Excellent Integrity Rating
         

[To Do:] Next tag candidates might still need to be assigned here:
size, signer, filename, Developer, Date, Author
truecolor?
audiochannels?
File format, Number of pages, Compilation, Description
Bandwidth?


D. KazaaNet Local SuperNode List Cache

FastTrack nodes provide several services. Nodes with stable bandwidths and enough upstream capacity, can - when allowed by the user - become supernodes. They then provide File Search and Network Management services to less-equipped client nodes. Client nodes connect to the FastTrack network with a single TCP-connection to a single supernode. When a FastTrack node needs to reconnect to this virtual P2P network, the availability and reachability of subsets of supernodes from a locally cached list of supernodes - typically with 200 entries - is probed. Reachable supernodes respond with updates of this Supernode List.

For probing, initially FastTrack protocol packets over UDP/IP are used. When no one responded to such an UDP-probe, connections over TCP/IP are tried next.

Candidate FastTrack "connection point" supernodes are selected from a dynamic SuperNode list that survives application terminations. The local supernode list cache is persisted in the Windows Local Machine Registry at HKLM\Software\KaZaA\ConnectionInfo\KazaaNet. Each time a new connection to the P2P network is (re)established - and sometimes (mostly after a Search Request) in between - some supernode entries in the list are dynamically replaced to reflect the more recent state. Supernodes send partial updates to clients and exchange updates between themselves. A (former) supernode may respond to a network connection request of a client with the advise to try to TCP-connect to another supernode ("redirect" the client connection request).

I did not see the Kazaa 2.[0,1] application in client mode try to connect to a supernode using any hardcoded IP-address. At each client-mode application start, the attempt to connect to the virtual P2P network fails when all probes to all supernodes cached in the registered supernode list fail. The user interface does not show such an unable to connect state though, it keeps saying "Connecting...".
Starting up in (forced full) supernode mode needs more research however. Joltid Ltd. - supplier of the FastTrack protocol stack code - (claims to) operate(s) so-called
kaasupernode-s at the top level of the 3-layer network. These top nodes effectively control the network of peers that are allowed to be SuperNodes. The Kazaa application also seems to contain code to support an Exclusion list.

Normal FastTrack nodes (clients) tend to be only known in a local "small world" (within their ISP-network, country, continent). Their FastTrack network "connection point" is a nearest supernode, most recently registered by the network, having the lowest workload percentage. Their FastTrack world gets smaller each time they (re)connect and search because of supernodelist replication based on "distance" timings. The distances appear to get occasionally measured using "firewall testing" protocol handshakes, that are performed on the request of (other) supernodes. Search Requests might be completely satifistied by the near supernodes.

This works great to limit FastTrack traffic on the global Internet, but your Search Results tend to be localised too. Only a small percentage of supernodes in other small worlds seems to get (randomly ?) injected in the supernode list by the automated FastTrack P2P network management. Joltid Ltd sells PeerCache server software for ISP-s with which they are able to streamline the FastTrack-traffic in their subnets. The KaZaA 2.[0,1] application supports such CacheHost-s with a custom Cache Discovery protocol (/.p2pcache).

Any initial standard "traceroute" ICMP pinging is only used to determine up- and downstream bandwidths and probably to optimize distance measurements too. The bandwith measurement traceroutes are always directed to four - pretty randomly chosen, non-FastTrack - IP-addresses in four hardcoded IP-subnets, likely assumed to be in four different continents of the global Internet:

204.152.*.* North America
130.244.*.* Europe - Scandinavia
202.139.*.* Australia
202.232.*.* Asia - Japan

With the traceroute pings, the "common hop" from the local FastTrack machine to the world is determined. This way the node's relative location is detected and the selection of supernode probe candidates from the locally cached supernodelist can be optimised against a distance parameter too.

Brilliant Digital Altnet's TopSearch add-on does contain a hardcoded IP-address (media.altnet.com). On startup, it makes the Kazaa 2.[0,1] client retrieve a TopSearchIndex file with this URL:

http://217.116.227.249:80/topsearch/tsi2.cab

This MS Authenticode signed CABinet file (with encrypted contents) likely has nothing to do with FastTrack network management, and probably only contains predefined Search Hits to payed content.

The KazaaNet list is a sequence of fixed length SuperNode items. The list is prefixed with a single byte. The number of items is variable. In a Kazaa 2.[0,1] client typically 200 supernodes are and/or will get registered. If the list is empty or no supernode was reachable (no one responded to an UDP-probe), the application's TCP-connection attempt to the FastTrack P2P network (any single supernode) will fail.

Table 4: KazaaNet Local Supernode List Cache [Persisted in Windows Registry]:

# Name Type Values Description
1 netListVersion BYTE 1 Prefix of the list with a variable number of next fixed length SuperNode items. Seems to be 1 always. Old Morpheus had 0 here.
2

Each supernode item subsequence is a fixed length entry, with this layout:

1 snIPnum DWORD 0x01020304 = 0x04,0x03,0x02,0x01 = "1.2.3.4." 32-bit IP-address. All other things equal, the client seems to prefer to (re)connect to supernodes having the most matching bits in their IP-numbers. "Nearness" could be estimated by comparing fully qualified domain names too.
2 snPort WORD 1831 UDP (and/or TCP?) File Search (and Network Management ?) service Port. KAZAA ListenPort.
3 snLoad BYTE 54 Percentage (0..99), reflecting the supernode's current workload/networkfullness. Can be 0. The client prefers to (re)connect to supernodes with low snLoad values.
4 snAvailability BYTE 0 = candidate, 1 = being probed ?, 2 = dead/unreachable ? The current state of the supernode. Mostly 0. Locally updated by client too.
5 snCreationTime DWORD 0x21,0x49,0x17,0x3E (= 2003-01-04T21:50:41Z) Datetime in seconds since Epoch (1970) that this record was created in the supernode network ? Or that this supernode was probed (by another node) ? The client prefers to (re)connect to supernodes with more recent registration times. Does the least significant byte hold some bits of a (4 ?) world-grid cell number indication for neighbour detection ?

End of supernode item subsequence.


I could not have created this page without the hacking performed by: Random Nut, idleloop, vlaibb, Sanderson, mospum, Scott J., Hasnain S., and everyone else I forgot to mention here. Thank you all. Keep sharing your information.

By: FdJ, Hrl, Nl Date: 2003-01-05T12:54:03+01:00
Last Mods:   Date: 2005-08-01T22:45:00+02:00

Pageloads from 2003-04-25T08:00:00+02:00 to 2005-08-01T22:45:00+02:00