6. Overview UPDF Files

UPDF (Uni-Plot-Data-Format) will be the new file format for UniPlot R2012. It will be used for UniPlot document files (IPZ files) and netCDF files (NC2 files).

UniPlot R2012 will still be able to create and read the original netCDF format. Both formats are using the same netCDF interface. A flag will indicate the new format during file creation.

6.1. Format Description

UPDF files are set up like file systems. An UPDF file contains directories. The directories can contain files and directories. As in a file system, files can be added, removed, renamed, extended or moved.

../../_images/overview-updf-de.png

Directories and files can contain attributes, so called key-value pairs. Different data types, e.g. Unicode strings or 64-bit integers, are supported. Scalar values, vectors and matrices of values can be saved.

To avoid confusion directories are named storages, files are named streams and attributes are named properties. Properties have a property name and a property value.

6.2. Advantages of the new Format

  • Writing UniPlot 5.x files can be very slow for big files with multiple pages. Writing of UPDF files is faster: The time to write a file increases linearly, i.e. to write 200 pages takes only twice as long as 100 pages.

  • The current IPW files and nc files do not use compression. UPDF uses compression and thus data is saved compactly.

  • Current nc files are limited to 2 Giga-Bytes in size. UPDF files are limited to 8 Tera-Bytes.

  • NC-files encounter performance problems if data is modified, e.g. if a global attribute has been added to a file, the complete nc file must be written to disk. This is not necessary for NC2 files.

  • Transactions are possible for NC2 files.

6.3. Format Properties

  • Max. size of UPDF files: approx. 8 Terabytes (8000 GB).

  • Number of characters for storage-, stream- and property names: 255.

  • Max. size of streams: 8 TB.

  • Max. number of streams and storages per storage: 2^31. In real life some 10,000.

  • Max. size of a property value: Some MB’s (must fit in RAM memory).

  • Number of properties per stream or storage: 2^31. In real life some 100,000.

6.4. Valid Names in Storages, Streams and Properties

A valid names contains 1 to 255 Unicode characters.

The following characters are invalid: Characters in in range 0 to 31 and the following 9 characters ‘/’, ‘\’, ‘*’, ‘?’, ‘”’, ‘<’, ‘>’, ‘:’, ‘|’.

6.5. Logging

Normally UPDF function errors are written in UniPlot’s Event Window. With the command:

log_set_active("updf", 0)

logging can be turned off.

6.6. Garbage collection

Garbage collection is used for all pointers. Unclosed elements will be closed in the correct order by the garbage-collector (gc).

6.7. UniScript Interface

Tools

updf_dump

updf_dump saves a complete updf file in a directory. For storages a new subdirectory will be created. Properties will be saved in xml files. Streams will be saved in binary files.

updf_fileversion

updf_fileversion returns the file version.

updf_gen

updf_gen copies the all files of directory an its subdirectories into an updf file.

updf_get_error_string

updf_get_error_string returns the error message for the error code returned by updf_is_error.

updf_get_filetimes

updf_get_filetimes returns the file times of an open UPDF file.

updf_get_parent

updf_get_parent returns the parent storage of a stream or storage.

updf_is_error

updf_is_error checks if a fatal error occurred.

updf_libversion

updf_libversion.

updf_set_filetimes

updf_set_filetimes sets the file times of an open UPDF file.

Storages and Streams

updf_close_storage

updf_close_storage closes a storage of sub storage.

updf_commit

If a root storage is given the changes are written to disk.

updf_copy_to

updf_copy_to copies a storages and all containing streams, storages and properties recursively into another storage.

updf_create_file

updf_create_file creates a new updf file.

updf_create_storage

updf_create_storage creates a new storage within a storage.

updf_create_stream

updf_create_stream creates a new stream in a storage.

updf_destroy_element

updf_destroy_element removes (destroys) a storage or stream in a storage.

updf_get_cmode

updf_get_cmode return the mode the UPDF file has been created with.

updf_get_free_pages

updf_get_free_pages returns the number of unused pages in a UPDF file.

updf_open_file

updf_open_file opens an existing updf file.

updf_open_storage

updf_open_storage opens a storage in a updf file.

updf_open_stream

updf_open_stream opens an existing stream in a updf file.

updf_rename_element

updf_rename_element changes the name of a stream or storage.

updf_revert

updf_revert reverts all modifications since the last open or commit command (updf_commit).

Streams

updf_create_stream

updf_create_stream creates a new stream in a storage.

updf_file_close

updf_file_close closes a updf stream.

updf_file_copy_to

updf_file_copy_to.

updf_open_stream

updf_open_stream opens an existing stream in a updf file.

Iteration (Storages and Streams)

updf_enum_begin

updf_enum_begin can be used with updf_enum_next to iterate over all streams and storages of a storage object.

updf_enum_close

updf_enum_close

updf_enum_next

updf_enum_next can be used with updf_enum_begin to iterage over all streams and storages of a storage object.

updf_stat_get_len

updf_stat_get_len is the size of a stream (number of bytes). If stat contains a storage object the function returns 0.

updf_stat_get_name

updf_stat_get_name returns the storage of stream name.

updf_stat_get_type

updf_stat_get_type returns the type of an element.

Properties

updf_prop_close

updf_prop_close closes the properties in a storage or stream.

updf_prop_copy_to

updf_prop_copy_to.

updf_prop_count

updf_prop_count returns the number of properties.

updf_prop_delete

updf_prop_delete removes a property-object (key/value pair).

updf_prop_enum

updf_prop_enum iterates over all properties of a stream or storage.

updf_prop_enum_close

updf_prop_enum_close closes a propEnum object.

updf_prop_enum_next

updf_prop_enum_next can be used with updf_prop_enum to iterate over all properties of a stream and storage.

updf_prop_get_at

updf_prop_get_at returns a property-object.

updf_prop_has_key

updf_prop_has_key checks if the given property-name exists.

updf_prop_open

updf_prop_open opens the properties of a given storage or stream.

updf_prop_set_at

updf_prop_set_at writes a property.

Property-Values

updf_propval_get

updf_propval_get returns a property value.

updf_propval_get_info

updf_propval_get_info returns information about a property.

updf_propval_new

updf_propval_new creates a new property value object.

id-824069