7. Overview UniPlot Data Files¶

If you import data into UniPlot, the data will be saved in an UniPlot data file format (nc2).

Up until UniPlot version R2011, UniPlot could create netCDF files. The standard extension is .nc. netCDF is a file format for measured data developed by the University Corporation for Atmospheric Research (see Copyright netCDF). Beginning with UniPlot R2012, netCDF files are not used to load data.

7.1. Content of a UniPlot Data File (nc2)¶

The following listing contains the output of ncdump.exe. The program lists the contents of a nc- or nc2 file as a text file. ( ncgen.exe can be used to convert the text file into a nc2 file.)

netcdf Test1.xls {
dimensions:
   n = 5 ;
variables:
       float EngSpd(n) ;
             EngSpd:title = "EngSpd" ;
             EngSpd:long_name = "EngSpd [RPM]" ;
             EngSpd:units = "RPM" ;
       float PME(n) ;
             PME:title = "PME" ;
             PME:long_name = "PME [bar]" ;
             PME:units = "bar" ;

// global attributes:
         :Origin = "D:\\uniplot\\samples\\Test1.xls" ;
         :Source = "STS Software and Testing Solutions GmbH (www.uniplot.de)" ;
         :Creator = "UniPlot Excel Converter v3" ;

data:

  EngSpd = 1006.00, 1249.00, 1512.00, 1708.00, 1804.00;
  PME = 8.47, 9.33, 10.64, 11.21, 11.27;
}

7.1.1. Dimensions¶

Each channel has a named dimension. All channels with the same dimension are displayed together in a table in the Data Editor.

If the channel A has the dimension n1 = 10 (10 points) and the channel B has the dimension n2 = 10 (also 10 points), the channels will be displayed in the data editor in two two different tables.

A channel can also contain several dimensions, but UniPlot uses this property only for string channels. The first dimension is the number of strings and the second dimension is the length of the longest string. String channels are a matrix of characters.

7.1.2. Attributes¶

An attribute is a key/value pair. The key is a name and the value element is a scalar number, or a vector of numbers, or a string.

UniPlot data files supports the following data types for channels and attributes:

Data Type	Meaning
NC_CHAR	Character (8 bit).
NC_INT8	Integer numbers with 8 bits (1 Byte). (Range -127 to 128 or 0 to 255.)
NC_INT16	Integer numbers with 16 bits (2 Bytes).
NC_INT32	Integer numbers with 32 bits (4 Bytes).
NC_FLOAT	Real number with 32 bits (4 Bytes).
NC_DOUBLE	Real numbers with 64 bits (8 Bytes).

Besides the data type, an attribute has a length. If the data type is NC_CHAR, the length is the number of bytes. For all other types the length specifies the number of values in the attribute.

The attribute can only be accessed as a complete vector. For instance, to modify a character in a string, the complete string must be read, modified and written back.

The size of an attribute should be small (less than 1,000 bytes).

A UniPlot data file can contain up to 20,000 global attributes. Global is the name of the attributes because they refer to the whole file. The channels also contain a list of attributes, the channel attributes.

An UniPlot data file can contain an unlimited number of attributes. A file can contain channel attributes and global attributes.

See Standard Attributes for more about writing attributes.

7.1.3. Channel¶

Channels are also Key/Value pairs. The key is the channel name and the value is a vector of numbers. The vector can contain up to 2.1 billion elements.

time = [0.1, 0.2, 0.3, ..., 2.000.000]

7.2. Channel Names¶

Channel names may contain the following special characters: ., -, +, $, #, ~, !, ^, &, %. To avoid problems with the formula interpreter, data exchange or compatibility problems, the channel name can be used with underscores instead of the special characters. Example: A NC-file contains a the channel AI50%+m. The following two calls will return the correct, identical varid:

varid = nc_varid(ncid, "AI50%+m");
varid = nc_varid(ncid, "AI50__m");

If the NC file contains channel names which only differ in special characters the function may return different varids. The reason is that the function returns the first channel id that matches the channel name. Example: A NC file contains the following two channels AI50%+m (varid 0) and AI50%-m (varid 1):

The following call will return the varid = 1:

varid = nc_varid(ncid, "AI50%-m");

If called with two underscores the function returns the varid of AI50%+m, varid = 0:

varid = nc_varid(ncid, "AI50__m");

To avoid problems, all channel names should be unique even if the special characters are replaced by underscores. If the NC files are created with UniPlot, the channel names which are only different in the special characters are enumerated, starting with 0. The first channel name will be unmodified.

7.3. Properties of UniPlot data files (nc2)¶

Usually, when editing a data file (adding channels, remove channels) the results for most data formats is writing a modified copy of the file. This is not necessary for nc2 files. This enables much higher performances for these operations.
Stability: In netCDF nc_abort cannot be used to undo modifications. If a number of changes are executed and one modification fails because of power failure, a network problem or a software bug the complete file is destroyed. For nc2 files all modifications can be undone because the files use transactions.
Reduced file size, because parts of the file are saved compressed. The following table compares the file size of three typical MDF/INCA files with netCDF- and netCDF-up files (all values in kBytes):

File

Original (MDF/VS100)

netCDF

nc2

File small

11

17

12

File medium

934

1426

453

File big

13727

33883

4271

Sometimes (for example pure random numbers) nc2 files can be bigger than netCDF fields. This will be a rare case.
No Limitation in size - no 2 GBytes limit. File size is limited to 1 TB. Channel size is limited to 2.1 billion data points.

7.4. Functions¶

	Tools
NC_AddFiles	NC_AddFiles merges NC files into a new NC file by adding data records to the destination file. Channels with identical names will be concatenated.
NC_Classify	NC_Classify classifies the data into groups. For each group an index channel is created. The index channel starts with the value 1 and is constant for each group. The second group receives the index value 2 and so on. If invoked with more than 2 parameters the data file can be split into single files depending on the given index channel name.
NC_CreateMeanCycleFile	NC_CreateMeanCycleFile creates a new NC file with the mean, minimum and maximum cycle of an NC data file containing multiple cycles (segments).
NC_CreateSampleReduction	NC_CreateSampleReduction creates a file with all channels with mor than 2 million data points in the source file.
NC_DeleteFilteredRecords	NC_DeleteFilteredRecords remove all filtered records from the given NC file.
NC_Edit	NC_Edit opens a dialog box to edit a UniPlot data file (NC file).
NC_ExportData	NC_ExportData converts UniPlot data files (.nc, .nc2) into another data format.
NC_GetVarAttribs	NC_GetVarNames returns a sgtring matrix with channel attribute values.
NC_GetVarNames	NC_GetVarNames returns the variable names of a netCDF data file (NC file).
NC_Interpolation	NC_Interpolation creates a new NC file with interpolated data. The function can be used to change the sampling rate (frequency resolution). MDF/Inca files with multiple time channels can be interpolated onto a common time channel.
NC_MapInterpolation	NC_MapInterpolation calculates new channels by map interpolation and adds the new channels to an NC file.
NC_MergeFiles	NC_MergeFiles merges NC files into a new NC file by adding the channels to the destination file.
nc_convert_units	nc_convert_units convert the channel units to SI units or common units.
nc_from_obj	nc_from_obj creates a netCDF file from a special structured UniScript object.
nc_to_obj	nc_to_obj reads a netCDF file or a selection of channels of a netCDF file into a UniScript object.

	Open and Close
nc_abort	nc_abort closes or deletes a data file.
nc_close	nc_close closes an open nc file.
nc_create	nc_create creates a new netCDF file.
nc_open	nc_open opens an existing data file.
nc_sync	nc_sync writes all buffered data to the file.

	Global Functions
nc_copy	nc_copy copies all variables and attributes of an NC file to a new NC file.
nc_diminq_size	nc_diminq_size returns the size of a dimension when given its ID.
nc_endef	nc_endef takes an open nc file out of define mode. The changes made to the nc file while it was in define mode are checked and committed to disk if no problems occurred. In particular, non-record variables are filled with their fill-values unless nc_setfill has been called with the argument `NC_NOFILL`. The nc file is then placed in data mode, so that variable data can be read or written.
nc_fatal_error	nc_fatal_error.
nc_filename	nc_filename returns the file name for a given ncid.
nc_get_enum_values	nc_get_enum_values return a matrix with enum values or an empty string.
nc_get_option	nc_get_option gets an option.
nc_inquire_format	nc_inquire_format returns the file format for a given ncid.
nc_inquire_mode	nc_inquire_mode returns the mode of an open nc-file.
nc_inquire_ndims	nc_inquire_ndims returns the number of dimensions in a nc file.
nc_inquire_ngatts	nc_inquire_ngatts returns the number of global attributes in a nc file.
nc_inquire_nvars	nc_inquire_nvars returns the number of variables in a nc file.
nc_inquire_recdim	nc_inquire_recdim returns the ID number of the record variable in a nc file.
nc_last_error	nc_last_error returns the last error that occurred of UniPlot data files. If the last invoked `nc_`-function returned without an error the function will return 0.
nc_redef	nc_redef puts an open netCDF file into define mode, so dimensions, variables, and attributes can be added or renamed and attributes can be deleted.
nc_set_option	nc_set_option sets an option.
nc_seterror_options	nc_seterror_options sets the error options for the netCDF functions.
nc_setfill	nc_setfill determines whether or not variable prefilling will be done.
nc_str_error	nc_str_error.

	Dimensions
nc_dimdef	nc_dimdef adds a new dimension to an open netCDF file.
nc_dimid	nc_dimid returns the ID of a netCDF dimension when given the name of the dimension.
nc_diminq_name	nc_diminq_name returns the name of a dimension when given its ID.
nc_dimredim	nc_dimredim modifies the size of a dimension. The NC2 file must be set to define mode (see nc_endef/nc_redef).
nc_dimrename	nc_dimrename renames a given dimension.

	Variables
nc_makevalidname	nc_makevalidname returns a valid netCDF variable name for a given name.
nc_varcopy	nc_varcopy copies the data of a variable from one NC file to another NC file. The variable must exist in both files and must have the same number of points.
nc_vardef	nc_vardef adds a new variable to an open netCDF file in define mode.
nc_vardelete	nc_vardelete deletes a given variable. A variable can only be deleted if the file is not in definition mode.
nc_varget	nc_varget reads data values from a netCDF variable of an open netCDF file. The file must be in data mode.
nc_varget_missing	nc_varget_missing reads data values from a netCDF variable of an open netCDF file. The file must be in data mode.
nc_varid	nc_varid returns the ID of a netCDF variable when given its name.
nc_varinq_changed	nc_varinq_changed returns the change counter of a netCDF variable.
nc_varinq_datatype	nc_varinq_datatype returns the data type of a variable when given its ID.
nc_varinq_dimids	nc_varinq_dimids returns a vector of dimension IDs.
nc_varinq_info	nc_varinq_info returns information about a channel. (Minimum, Maximum, Monotonic, channel contains missing values).
nc_varinq_name	The nc_varinq_name function returns the name of a netCDF variable when given its ID.
nc_varinq_natts	The nc_varinq_natts function returns the number of attributes for a variable, given its ID.
nc_varput	The nc_varput functions writes data values into a netCDF variable of an open netCDF file. The file must be in data mode.
nc_varput_missing	The nc_varput_missing functions writes data values into a netCDF variable of an open netCDF file. The file must be in data mode. Missing values must have the value `MISSING_VALUE`.
nc_varput_text	The nc_varput_text functions writes string data values into a netCDF variable of an open netCDF file. The file must be in data mode.
nc_varrename	nc_varrename renames a given netCDF variable.
nc_varsearch	nc_varsearch searches in a given variable for the index of that value that is greater of equal than x. The values in variable must be sorted monotonically increasing and should not contain missing values.

	Attributes
nc_attcopy	nc_attcopy copies a given attribute into a different NC file.
nc_attdelete	nc_attdelete deletes a given attribut. The file must be in define mode.
nc_attget	nc_attget returns the value(s) of a netCDF attribute.
nc_attinq_datatype	nc_attinq_datatype returns the data type of an attribute when given its ID and name.
nc_attinq_len	nc_attinq_len returns the number of values stored in the attribute. If the attribute is of type `NC_CHAR`, the function returns the string length.
nc_attname	nc_attname returns the attribute name.
nc_attput	nc_attput changes a variable attribute or global attribute of an open nc file. If the attribute is new, or if the space required to store the attribute is greater than before, the nc file must be in define mode.
nc_attrename	nc_attrename renames a given attribute.

7.5. Standard Attributes¶

The netCDF file format is UniPlot’s standard data file format. These files can contain attributes of the form

AttributName = AttributValue

A netCDF attribute contains information about a netCDF variable or about the entire netCDF file. Attributes are used to specify such properties as units, special values and parameters. For example, “units” is an attribute represented by a string such as “Nm”

The attribute names are case sensitive. Units and units are different attributes.

Here is a list of attribute names that have a special meaning for UniPlot:

7.5.1. Global Attributes¶

Attribute Name	Data Type	Meaning
Origin	char	Name of a source file, e.g. “c:test.xls”.
Source	char	Import filter creator.
Creator	char	Name of the import filter. The name may contain a version number, e.g. UniPlot Pasy 1.0.
Date	char	Date when the data file was created.
Time	char	Time when the data file was created.
Range	2*int
_nc_dl_driver, _nc_dl_driverinfo, _nc_dl_source, _nc_dl_loadcount	char and int	see Delayed Loading.

7.5.2. Channel Attributes¶

Attribute Name	Data Type	Meaning
title	char	A succinct description.
long_name	char	A long descriptive name containing the unit.
units	char	A string that specifies the units of the variable’s data.
scale_factor	double	If present, it is used to be multiplied by this factor after the data is read.
add_offset	double	If present, it is used to be added to the data after the data is read.
missing_value	same as the channel	The data type of the attribute missing_value is identical to the channel’s data type.
Type	char	Values is “Time Channel” or “Data Channel”
datatype	char	Values are: “date”, “time”, “datetime”
C_format	char	Format string that can be used to print values, see printf.
__Delete__	int	Channel is marked as deleted if the value is 1.
Description	char	A description.
Comment	char	A comment.
_FillValue
_formula_text	char
_formula_description	char
_formula_comment	char
_formula_onlyif	char
_channel_type	char	“formula” or “formula_not_registered”.
valid_range	same as the channel	`valid_range` is a channel attribute that holds two numbers specifying the minimum and maximum valid values. The type of each `valid_range` attribute should match the type of its variable. If a channel value is outside the valid range, it will be displayed in the grid with the font color red.
_nc_enum	char	The attribut contains value/text pairs. The elements are separated by an Or character (\|). Example: `0\|Off\|1\|On\|2\|Error`. The order is `Value\|Text\|Value\|Text\|...`. The decimal separator is a point (.). See also nc_get_enum_values. If the attribut is available, it will be used as user labels for the y axis.
XStart_XDelta	double[2]	Start value and delta. Used, if a channel is used for a t/y dataset.
nc_min, nc_max, nc_monotone, nc_has_missings, nc_change_counter	double	See nc_varinq_info.
_nc_dl_loaded, _nc_dl_loadinfo	int	see Delayed Loading.

7.6. Channel Grouping (Tree Structure)¶

To display a channel in a group the attribute _nc_group (data type NC_CHAR) must be added to the channel, e.g. _nc_group = "Angle Data/Heat Release" or _nc_group = "G1". To enable the groups create a global attribute _nc_hasgroups = 1 (NC_LONG):

nc_attput(ncid, -1, "_nc_hasgroups", NC_BYTE, 1);
nc_attput(ncid, varid, "_nc_group", NC_CHAR, "G1");

The grouping will be displayed it the tree structur button is pressed. The Browser_ShowTreeStructure() function toggles this button. The return value 1 means the function is enabled. The global variable _g().nc_use_groups saves the setting.

7.7. Delayed Loading¶

If the “Delayed Channel Import” is enabled, (see Tools=>More Options, Group: Data Import and Data Browser) the data channels will be loaded from the source file into the UniPlot data file when a channel is first accessed. Example: If a file contains 400 channels and only 8 channels are loaded into an IPW document the import is 50 times faster.

Global Attributes

_nc_dl_driver = "rs_usdata"
_nc_dl_driverinfo = "TDM"
_nc_dl_source = "d:\\test.tdm"
_nc_dl_loadcount = 0

Channel-Attributes

_nc_dl_loaded = 0
_nc_dl_loadinfo = "5,2,amp"

context = _XXX_DL_open(ncid, ssSource)
bool = _XXX_DL_load_channel(context, varid, ssInfo)
_XXX_DL_close(context)

Example:

def _TDM_DL_open(ncid, ssSource)
{
    ctx = [.];
    hTDMFile = _DDC_OpenFileEx(ssSource, "", TRUE /* READONLY */);
    if (type(hTDMFile) != "error") {
        ctx.hTDMFile = hTDMFile;
        ctx.ncid = ncid;
        ctx.ssOurce = ssSource;
        return ctx;
    }
    return 0;
}
def _TDM_DL_load_channel(ctx, varid, ssInfo)
{
    sv = strtok(ssInfo, ",");
    iGroup = strtol(sv[1]);
    iChannel = strtol(sv[2]);
    oGroups = _DDC_GetChannelGroups(ctx.hTDMFile);
    oChannels = _DDC_GetChannels(oGroups[iGroup]);
    nBlocksize = 10000;
    nStart = 0;
    nRead = nBlocksize;
    while (nRead >= nBlocksize) {
        rvData = _DDC_GetDataValues(oChannels[iChannel], nStart, nBlocksize);
        if (type(rvData) == "error") {
            return FALSE;
        }
        nRead = len(rvData);
        r = nc_varput(ctx.ncid, varid, nStart, nRead, rvData);
        nStart = nStart + nRead;
    }
    return TRUE;
}
def _TDM_DL_close(ctx)
{
    _DDC_CloseFile(ctx.hTDMFile);
}

7.8. Example¶

The following function creates a netCDF file. In this example the data is passed as a string matrix (smData).

def NC_WriteSimpleNCFile(ssFileName, ssNetCDFName, svChanName, svUnits, ...
            rvDataType, smData, ssCreator)
{
    nCols = nc(smData);
    nRows = nr(smData);
    ncid = nc_create(ssNetCDFName);
    if (ncid == -1) {
        ssError = sprintf(_s("Cannot create file %s - Is file already open?");
        MessageBox(ssError, ssNetCDFName), "ICONSTOP");
        return "#IMPORTERROR#";
    }
    dimids = nc_dimdef(ncid, "nRows", nRows);
    // Globale Attribute:
    nc_attput(ncid, -1, "Origin", NC_CHAR, _StandardFileName(ssFileName));
    nc_attput(ncid, -1, "Source", NC_CHAR, "STS Software and Testing Solutions GmbH");
    nc_attput(ncid, -1, "Creator", NC_CHAR, ssCreator);
    rvRange = [1, nRows];
    nc_attput(ncid, -1, "Range", NC_LONG, rvRange);
    nc_attput(ncid, -1, "Reference", NC_CHAR, "");
    nc_attput(ncid, -1, "Date", NC_CHAR, sum(date()));
    nc_attput(ncid, -1, "Time", NC_CHAR, sum(time()));
    nc_attput(ncid, -1, "Title", NC_CHAR, "");
    nc_attput(ncid, -1, "Comment1", NC_CHAR, "");
    nc_attput(ncid, -1, "Comment2", NC_CHAR, "");
    nc_attput(ncid, -1, "Comment3", NC_CHAR, "");
    for (i in 1:nCols) {
        if (NC_DATETIME == rvDataType[i]) {
            varid = nc_vardef(ncid, svChanName[i], NC_DOUBLE, dimids);
        } else {
            varid = nc_vardef(ncid, svChanName[i], rvDataType[i], dimids);
        }
        nc_attput(ncid, varid, "title", NC_CHAR, svChanName[i]);
        nc_attput(ncid, varid, "units", NC_CHAR, svUnits[i]);
        nc_attput(ncid, varid, "long_name", NC_CHAR, svChanName[i] + ..
                               "  [" + svUnits[i] + "]");
        nc_attput(ncid, varid, "scale_factor", NC_DOUBLE, 1.0);
        nc_attput(ncid, varid, "add_offset", NC_DOUBLE, 0.0);
        nc_attput(ncid, varid, "Description", NC_CHAR, "");
        nc_attput(ncid, varid, "Comment", NC_CHAR, "");
        nc_attput(ncid, varid, "ChanType", NC_CHAR, "DateTime");
    }
    nc_setfill(ncid, NC_NOFILL);
    nc_endef(ncid);
    for (i in 1:nCols) {
        if (rvDataType[i] == 100) {
            rvData = DT_ParseDateTime(smData[;i]);
        } else {
            rvData = strtod(smData[;i]); // string to double
        }
        nc_varput(ncid, i-1, 0, nRows, rvData);
    }
    nc_close(ncid);
    return ssNetCDFName;
}

7.9. Copyright netCDF¶

See netCDF and http://www.unidata.ucar.edu/software/netcdf/