6. Overview UPDF Files

UPDF (Uni-Plot-Data-Format) will be the new file format for UniPlot R2012. It will be used for UniPlot document files (IPZ files) and netCDF files (NC2 files).

UniPlot R2012 will still be able to create and read the original netCDF format. Both formats are using the same netCDF interface. A flag will indicate the new format during file creation.

6.1. Format Description

UPDF files are set up like file systems. An UPDF file contains directories. The directories can contain files and directories. As in a file system, files can be added, removed, renamed, extended or moved.

Directories and files can contain attributes, so called key-value pairs. Different data types, e.g. Unicode strings or 64-bit integers, are supported. Scalar values, vectors and matrices of values can be saved.

To avoid confusion directories are named storages, files are named streams and attributes are named properties. Properties have a property name and a property value.

6.2. Advantages of the new Format

• Writing UniPlot 5.x files can be very slow for big files with multiple pages. Writing of UPDF files is faster: The time to write a file increases linearly, i.e. to write 200 pages takes only twice as long as 100 pages.

• The current IPW files and nc files do not use compression. UPDF uses compression and thus data is saved compactly.

• Current nc files are limited to 2 Giga-Bytes in size. UPDF files are limited to 8 Tera-Bytes.

• NC-files encounter performance problems if data is modified, e.g. if a global attribute has been added to a file, the complete nc file must be written to disk. This is not necessary for NC2 files.

• Transactions are possible for NC2 files.

6.3. Format Properties

• Max. size of UPDF files: approx. 8 Terabytes (8000 GB).

• Number of characters for storage-, stream- and property names: 255.

• Max. size of streams: 8 TB.

• Max. number of streams and storages per storage: 2^31. In real life some 10,000.

• Max. size of a property value: Some MB’s (must fit in RAM memory).

• Number of properties per stream or storage: 2^31. In real life some 100,000.

6.4. Valid Names in Storages, Streams and Properties

A valid names contains 1 to 255 Unicode characters.

The following characters are invalid: Characters in in range 0 to 31 and the following 9 characters ‘/’, ‘\’, ‘*’, ‘?’, ‘”’, ‘<’, ‘>’, ‘:’, ‘|’.

6.5. Logging

Normally UPDF function errors are written in UniPlot’s Event Window. With the command:

log_set_active("updf", 0)

logging can be turned off.

6.6. Garbage collection

Garbage collection is used for all pointers. Unclosed elements will be closed in the correct order by the garbage-collector (gc).

6.7. UniScript Interface

	Tools
updf_dump	updf_dump saves a complete updf file in a directory. For storages a new subdirectory will be created. Properties will be saved in xml files. Streams will be saved in binary files.
updf_fileversion	updf_fileversion returns the file version.
updf_gen	updf_gen copies the all files of directory an its subdirectories into an updf file.
updf_get_error_string	updf_get_error_string returns the error message for the error code returned by updf_is_error.
updf_get_filetimes	updf_get_filetimes returns the file times of an open UPDF file.
updf_get_parent	updf_get_parent returns the parent storage of a stream or storage.
updf_is_error	updf_is_error checks if a fatal error occurred.
updf_libversion	updf_libversion.
updf_set_filetimes	updf_set_filetimes sets the file times of an open UPDF file.

	Storages and Streams
updf_close_storage	updf_close_storage closes a storage of sub storage.
updf_commit	If a root storage is given the changes are written to disk.
updf_copy_to	updf_copy_to copies a storages and all containing streams, storages and properties recursively into another storage.
updf_create_file	updf_create_file creates a new updf file.
updf_create_storage	updf_create_storage creates a new storage within a storage.
updf_create_stream	updf_create_stream creates a new stream in a storage.
updf_destroy_element	updf_destroy_element removes (destroys) a storage or stream in a storage.
updf_get_cmode	updf_get_cmode return the mode the UPDF file has been created with.
updf_get_free_pages	updf_get_free_pages returns the number of unused pages in a UPDF file.
updf_open_file	updf_open_file opens an existing updf file.
updf_open_storage	updf_open_storage opens a storage in a updf file.
updf_open_stream	updf_open_stream opens an existing stream in a updf file.
updf_rename_element	updf_rename_element changes the name of a stream or storage.
updf_revert	updf_revert reverts all modifications since the last open or commit command (updf_commit).

	Streams
updf_create_stream	updf_create_stream creates a new stream in a storage.
updf_file_close	updf_file_close closes a updf stream.
updf_file_copy_to	updf_file_copy_to.
updf_open_stream	updf_open_stream opens an existing stream in a updf file.

	Iteration (Storages and Streams)
updf_enum_begin	updf_enum_begin can be used with updf_enum_next to iterate over all streams and storages of a storage object.
updf_enum_close	updf_enum_close
updf_enum_next	updf_enum_next can be used with updf_enum_begin to iterage over all streams and storages of a storage object.
updf_stat_get_len	updf_stat_get_len is the size of a stream (number of bytes). If stat contains a storage object the function returns 0.
updf_stat_get_name	updf_stat_get_name returns the storage of stream name.
updf_stat_get_type	updf_stat_get_type returns the type of an element.

	Properties
updf_prop_close	updf_prop_close closes the properties in a storage or stream.
updf_prop_copy_to	updf_prop_copy_to.
updf_prop_count	updf_prop_count returns the number of properties.
updf_prop_delete	updf_prop_delete removes a property-object (key/value pair).
updf_prop_enum	updf_prop_enum iterates over all properties of a stream or storage.
updf_prop_enum_close	updf_prop_enum_close closes a propEnum object.
updf_prop_enum_next	updf_prop_enum_next can be used with updf_prop_enum to iterate over all properties of a stream and storage.
updf_prop_get_at	updf_prop_get_at returns a property-object.
updf_prop_has_key	updf_prop_has_key checks if the given property-name exists.
updf_prop_open	updf_prop_open opens the properties of a given storage or stream.
updf_prop_set_at	updf_prop_set_at writes a property.

	Property-Values
updf_propval_get	updf_propval_get returns a property value.
updf_propval_get_info	updf_propval_get_info returns information about a property.
updf_propval_new	updf_propval_new creates a new property value object.

See also

UniScript Functions Overview

_id-824069