MDF-Files can contain so called “unsorted” data blocks. A data block is unsorted, if it contains records of multiple channel groups. In a sorted MDF-File, each channel group has its own data block, only containing records for this specific collection of channels.
MDF4-Files can further contain other structures that may cause problems, or slow down the process of accessing the file contents. These structures are explained more detailed in the next section.
In an MDF-File each data section (either a single data block (DTBLOCK) or a list containing data blocks (DLBLOCK)) is referenced by a datagroup block (DGBLOCK). This datagroup block also references a list of channel groups (CGBLOCK). The records for these channels will be stored in the data group‘s data block. If a data group block’s channelgroup list consists of more than one CGBLOCK, it is unsorted, because the records of all the CGBLOCKs in the list will be stored in a single data section.
Such a “mixed” data block cannot be parsed by the ODS Server, and has therefore to be sorted in advance. Once this tool encounters a DGBLOCK with more than one channel group (CGBLOCK) it creates a new DGBLOCK (and the associated data section) for each channel group found. After processing, each DGBLOCK will only contain one channel group and a its data group will only contain the records of this channel group.
Normal channelgroups' records normally have a fixed length. This must not be the case for VLSD channelgroups. Theses groups contain data from a channel of another channelgroup, that has no specified fixed length. The variable length records are mixed into the data section with the other channel groups. In a sorted MDF-File, this is not allowed, because each channel group should contain only fixed length records. If a channel group contains a VLSD channel, the channel itself should have a link to a signal data block (SDBLOCK), containing the variable length records. Its fixed length part of the channel groups record should only contain offsets to the data in the signal data block. The contained data will be stored in a new SDBLOCK referenced in the corresponding channel.
An MDF4-File of version 4.1 or above can contain data compressed with the deflate algorithm. Such data is stored in a zipped data block (DZBLOCK), and cannot be read by the ODS Server. Blocks which can be compressed are data blocks (DTBLOCK), signal data blocks (SDBLOCK) and reduction data blocks (RDBLOCK). If the -zip flag is not set the output file will not contain any DZBLOCKS, and the data will be stored in an equivalent uncompressed block.
An MDF4-File may also contain a data section consisting of multiple blocks. Therefore the datalist block (DLBLOCK) is used, which stores an array of links to data blocks. Apart from that, it can also store a link to another datalist block with an array of data blocks and so on. This results in a linked list of data block arrays.
In general this tool tries to merge the data section into as few blocks as possible (size of blocks is limited by the maxblocksize-flag). If the whole section can be merged into one block the list block will be omitted. If a list block is needed, all data blocks in the list except the last one will contain maxblocksize bytes of data. This will also be specified in the maxblocksize attibute of the parent data list.
A data list is not touched, if splitting up the data into parts of maxblocksize bytes each would result in more blocks to be in the output, than were there before.
This tool provides two interfaces: A Command Line Interface (CLI), and an Java API, for usage in Java applications.
The JAR-File of this tool can be executed from the command line (use: java -jar .jar). The program needs the following arguments to work:
First, a general command has to be given. This can be either check, to see if processing a file is needed or process to resort a file.
process <Input file> <Output file> [<Flags>]
Input file: The path to the MDF3 / MDF4 file to be processed.
Output file: A valid file name, where the output should be written. If this file already exists, it is overwritten without prior warning.
Flags: The flags for this program call. The order of flags is not important. If none of these flags is passed, the program will run with its default values. The following flags are valid:
Example: process C:\file1.mf4 C:\file2.mf4 -unzip -maxblocksize=800k
check <Input file> <MaxBlockSize> [<ZipFlag>]
Input file: The path to the MDF3 / MDF4 file to be checked.
MaxBlockSize: The maximum size of data blocks in the output (in bytes). Default value is 2GB. The value can be passed using the decimal prefixes, e.g. 1k, 32m, 2g. Decimal values are not allowed.
ZipFlag: -zip or -unzip. If the -zip-flag is set the file is also checked for unzipped data. If it is not set, it is checked for zipped data. Default value: unzip. This flag is not compulsory.
Example: check C:\file1.mf4 800k -zip
The MDFSorter also provides a Java-Interface, for usage in other programs. It consists of two methods from the MDFSorter-Class:
Please refer to the JavaDoc documentation of the MDFSorter-Class for more detailed information, about arguments, semantics and return values.
The following limits of this tool are known:
Fixed NPE in MDF4ProcessWriter#fillRecordArray : NPE is caused by wrongly reading an UINT8 as INT8. The error is in MDF4Util#readUInt8 which is called by multiple other locations in the code. Fixing the method required changing the return type, which led to changes in various *BLOCKs.
Copyright (c) 2015-2018 Contributors to the Eclipse Foundation
See the NOTICE file(s) distributed with this work for additional information regarding copyright ownership.
This program and the accompanying materials are made available under the terms of the Eclipse Public License v. 2.0 which is available at http://www.eclipse.org/legal/epl-2.0.