6. Overview UPDF Files

UPDF (Uni-Plot-Data-Format) will be the new file format for UniPlot R2012. It will be used for UniPlot document files (IPZ files) and netCDF files (NC2 files).

UniPlot R2012 will still be able to create and read the original netCDF format. Both formats are using the same netCDF interface. A flag will indicate the new format during file creation.

6.1. Format Description

UPDF files are set up like file systems. An UPDF file contains directories. The directories can contain files and directories. As in a file system, files can be added, removed, renamed, extended or moved.

../../_images/overview-updf-de.png

Directories and files can contain attributes, so called key-value pairs. Different data types, e.g. Unicode strings or 64-bit integers, are supported. Scalar values, vectors and matrices of values can be saved.

To avoid confusion directories are named storages, files are named streams and attributes are named properties. Properties have a property name and a property value.

6.2. Advantages of the new Format

  • Writing UniPlot 5.x files can be very slow for big files with multiple pages. Writing of UPDF files is faster: The time to write a file increases linearly, i.e. to write 200 pages takes only twice as long as 100 pages.
  • The current IPW files and nc files do not use compression. UPDF uses compression and thus data is saved compactly.
  • Current nc files are limited to 2 Giga-Bytes in size. UPDF files are limited to 8 Tera-Bytes.
  • NC-files encounter performance problems if data is modified, e.g. if a global attribute has been added to a file, the complete nc file must be written to disk. This is not necessary for NC2 files.
  • Transactions are possible for NC2 files.

6.3. Format Properties

  • Max. size of UPDF files: approx. 8 Terabytes (8000 GB).
  • Number of characters for storage-, stream- and property names: 255.
  • Max. size of streams: 8 TB.
  • Max. number of streams and storages per storage: 2^31. In real life some 10,000.
  • Max. size of a property value: Some MB’s (must fit in RAM memory).
  • Number of properties per stream or storage: 2^31. In real life some 100,000.

6.4. Valid Names in Storages, Streams and Properties

A valid names contains 1 to 255 Unicode characters.

The following characters are invalid: Characters in in range 0 to 31 and the following 9 characters ‘/’, ‘', ‘*’, ‘?’, ‘”’, ‘<’, ‘>’, ‘:’, ‘|’.

6.5. Logging

Normally UPDF function errors are written in UniPlot’s Event Window. With the command:

log_set_active("updf", 0)

logging can be turned off.

6.6. Garbage collection

Garbage collection is used for all pointers. Unclosed elements will be closed in the correct order by the garbage-collector (gc).

6.7. UniScript Interface

  Tools
updf_dump updf_dump saves a complete updf file in a directory. For storages a new subdirectory will be created. Properties will be saved in xml files. Streams will be saved in binary files.
updf_fileversion updf_fileversion returns the file version.
updf_gen updf_gen copies the all files of directory an its subdirectories into an updf file.
updf_get_error_string updf_get_error_string returns the error message for the error code returned by updf_is_error.
updf_get_filetimes updf_get_filetimes returns the file times of an open UPDF file.
updf_get_parent updf_get_parent returns the parent storage of a stream or storage.
updf_is_error updf_is_error checks if a fatal error occurred.
updf_libversion updf_libversion.
updf_set_filetimes updf_set_filetimes sets the file times of an open UPDF file.
  Storages and Streams
updf_close_storage updf_close_storage closes a storage of sub storage.
updf_commit If a root storage is given the changes are written to disk.
updf_copy_to updf_copy_to copies a storages and all containing streams, storages and properties recursively into another storage.
updf_create_file updf_create_file creates a new updf file.
updf_create_storage updf_create_storage creates a new storage within a storage.
updf_create_stream updf_create_stream creates a new stream in a storage.
updf_destroy_element updf_destroy_element removes (destroys) a storage or stream in a storage.
updf_get_cmode updf_get_cmode return the mode the UPDF file has been created with.
updf_get_free_pages updf_get_free_pages returns the number of unused pages in a UPDF file.
updf_open_file updf_open_file opens an existing updf file.
updf_open_storage updf_open_storage opens a storage in a updf file.
updf_open_stream updf_open_stream opens an existing stream in a updf file.
updf_rename_element updf_rename_element changes the name of a stream or storage.
updf_revert updf_revert reverts all modifications since the last open or commit command (updf_commit).
  Streams
updf_create_stream updf_create_stream creates a new stream in a storage.
updf_file_close updf_file_close closes a updf stream.
updf_file_copy_to updf_file_copy_to.
updf_open_stream updf_open_stream opens an existing stream in a updf file.
  Iteration (Storages and Streams)
updf_enum_begin updf_enum_begin can be used with updf_enum_next to iterate over all streams and storages of a storage object.
updf_enum_close updf_enum_close
updf_enum_next updf_enum_next can be used with updf_enum_begin to iterage over all streams and storages of a storage object.
updf_stat_get_len updf_stat_get_len is the size of a stream (number of bytes). If stat contains a storage object the function returns 0.
updf_stat_get_name updf_stat_get_name returns the storage of stream name.
updf_stat_get_type updf_stat_get_type returns the type of an element.
  Properties
updf_prop_close updf_prop_close closes the properties in a storage or stream.
updf_prop_copy_to updf_prop_copy_to.
updf_prop_count updf_prop_count returns the number of properties.
updf_prop_delete updf_prop_delete removes a property-object (key/value pair).
updf_prop_enum updf_prop_enum iterates over all properties of a stream or storage.
updf_prop_enum_close updf_prop_enum_close closes a propEnum object.
updf_prop_enum_next updf_prop_enum_next can be used with updf_prop_enum to iterate over all properties of a stream and storage.
updf_prop_get_at updf_prop_get_at returns a property-object.
updf_prop_has_key updf_prop_has_key checks if the given property-name exists.
updf_prop_open updf_prop_open opens the properties of a given storage or stream.
updf_prop_set_at updf_prop_set_at writes a property.
  Property-Values
updf_propval_get updf_propval_get returns a property value.
updf_propval_get_info updf_propval_get_info returns information about a property.
updf_propval_new updf_propval_new creates a new property value object.

id-824069