7. Overview UniPlot Data Files

If you import data into UniPlot, the data will be saved in an UniPlot data file format (nc2).

Up until UniPlot version R2011, UniPlot could create netCDF files. The standard extension is .nc. netCDF is a file format for measured data developed by the University Corporation for Atmospheric Research (see Copyright netCDF). Beginning with UniPlot R2012, netCDF files are not used to load data.

7.1. Content of a UniPlot Data File (nc2)

The following listing contains the output of ncdump.exe. The program lists the contents of a nc- or nc2 file as a text file. ( ncgen.exe can be used to convert the text file into a nc2 file.)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
netcdf Test1.xls {
dimensions:
   n = 5 ;
variables:
       float EngSpd(n) ;
             EngSpd:title = "EngSpd" ;
             EngSpd:long_name = "EngSpd [RPM]" ;
             EngSpd:units = "RPM" ;
       float PME(n) ;
             PME:title = "PME" ;
             PME:long_name = "PME [bar]" ;
             PME:units = "bar" ;

// global attributes:
         :Origin = "D:\\uniplot\\samples\\Test1.xls" ;
         :Source = "FEV Software and Testing Solutions GmbH (www.uniplot.de)" ;
         :Creator = "UniPlot Excel Converter v3" ;

data:

  EngSpd = 1006.00, 1249.00, 1512.00, 1708.00, 1804.00;
  PME = 8.47, 9.33, 10.64, 11.21, 11.27;
}

7.1.1. Dimensions

Each channel has a named dimension. All channels with the same dimension are displayed together in a table in the Data Editor.

If the channel A has the dimension n1 = 10 (10 points) and the channel B has the dimension n2 = 10 (also 10 points), the channels will be displayed in the data editor in two two different tables.

A channel can also contain several dimensions, but UniPlot uses this property only for string channels. The first dimension is the number of strings and the second dimension is the length of the longest string. String channels are a matrix of characters.

7.1.2. Attributes

An attribute is a key/value pair. The key is a name and the value element is a scalar number, or a vector of numbers, or a string.

UniPlot data files supports the following data types for channels and attributes:

Data Type Meaning
NC_CHAR Character (8 bit).
NC_INT8 Integer numbers with 8 bits (1 Byte). (Range -127 to 128 or 0 to 255.)
NC_INT16 Integer numbers with 16 bits (2 Bytes).
NC_INT32 Integer numbers with 32 bits (4 Bytes).
NC_FLOAT Real number with 32 bits (4 Bytes).
NC_DOUBLE Real numbers with 64 bits (8 Bytes).

Besides the data type, an attribute has a length. If the data type is NC_CHAR, the length is the number of bytes. For all other types the length specifies the number of values in the attribute.

The attribute can only be accessed as a complete vector. For instance, to modify a character in a string, the complete string must be read, modified and written back.

The size of an attribute should be small (less than 1,000 bytes).

A UniPlot data file can contain up to 20,000 global attributes. Global is the name of the attributes because they refer to the whole file. The channels also contain a list of attributes, the channel attributes.

An UniPlot data file can contain an unlimited number of attributes. A file can contain channel attributes and global attributes.

See Standard Attributes for more about writing attributes.

7.1.3. Channel

Channels are also Key/Value pairs. The key is the channel name and the value is a vector of numbers. The vector can contain up to 2.1 billion elements.

time = [0.1, 0.2, 0.3, ..., 2.000.000]

7.2. Channel Names

Channel names may contain the following special characters: ., -, +, $, #, ~, !, ^, &, %. To avoid problems with the formula interpreter, data exchange or compatibility problems, the channel name can be used with underscores instead of the special characters. Example: A NC-file contains a the channel AI50%+m. The following two calls will return the correct, identical varid:

varid = nc_varid(ncid, "AI50%+m");
varid = nc_varid(ncid, "AI50__m");

If the NC file contains channel names which only differ in special characters the function may return different varids. The reason is that the function returns the first channel id that matches the channel name. Example: A NC file contains the following two channels AI50%+m (varid 0) and AI50%-m (varid 1):

The following call will return the varid = 1:

varid = nc_varid(ncid, "AI50%-m");

If called with two underscores the function returns the varid of AI50%+m, varid = 0:

varid = nc_varid(ncid, "AI50__m");

To avoid problems, all channel names should be unique even if the special characters are replaced by underscores. If the NC files are created with UniPlot, the channel names which are only different in the special characters are enumerated, starting with 0. The first channel name will be unmodified.

7.3. Properties of UniPlot data files (nc2)

  • Usually, when editing a data file (adding channels, remove channels) the results for most data formats is writing a modified copy of the file. This is not necessary for nc2 files. This enables much higher performances for these operations.

  • Stability: In netCDF nc_abort cannot be used to undo modifications. If a number of changes are executed and one modification fails because of power failure, a network problem or a software bug the complete file is destroyed. For nc2 files all modifications can be undone because the files use transactions.

  • Reduced file size, because parts of the file are saved compressed. The following table compares the file size of three typical MDF/INCA files with netCDF- and netCDF-up files (all values in kBytes):

    File Original (MDF/VS100) netCDF nc2
    File small 11 17 12
    File medium 934 1426 453
    File big 13727 33883 4271

    Sometimes (for example pure random numbers) nc2 files can be bigger than netCDF fields. This will be a rare case.

  • No Limitation in size - no 2 GBytes limit. File size is limited to 1 TB. Channel size is limited to 2.1 billion data points.

7.4. Functions

  Tools
NC_AddFiles NC_AddFiles merges NC files into a new NC file by adding data records to the destination file. Channels with identical names will be concatenated.
NC_Classify NC_Classify classifies the data into groups. For each group an index channel is created. The index channel starts with the value 1 and is constant for each group. The second group receives the index value 2 and so on. If invoked with more than 2 parameters the data file can be split into single files depending on the given index channel name.
NC_CreateMeanCycleFile NC_CreateMeanCycleFile creates a new NC file with the mean, minimum and maximum cycle of an NC data file containing multiple cycles (segments).
NC_CreateSampleReduction NC_CreateSampleReduction creates a file with all channels with mor than 2 million data points in the source file.
NC_DeleteFilteredRecords NC_DeleteFilteredRecords remove all filtered records from the given NC file.
NC_Edit NC_Edit opens a dialog box to edit a UniPlot data file (NC file).
NC_ExportData NC_ExportData converts UniPlot data files (.nc, .nc2) into another data format.
NC_GetVarAttribs NC_GetVarNames returns a sgtring matrix with channel attribute values.
NC_GetVarNames NC_GetVarNames returns the variable names of a netCDF data file (NC file).
NC_Interpolation NC_Interpolation creates a new NC file with interpolated data. The function can be used to change the sampling rate (frequency resolution). MDF/Inca files with multiple time channels can be interpolated onto a common time channel.
NC_MapInterpolation NC_MapInterpolation calculates new channels by map interpolation and adds the new channels to an NC file.
NC_MergeFiles NC_MergeFiles merges NC files into a new NC file by adding the channels to the destination file.
nc_convert_units nc_convert_units convert the channel units to SI units or common units.
nc_from_obj nc_from_obj creates a netCDF file from a special structured UniScript object.
nc_to_obj nc_to_obj reads a netCDF file or a selection of channels of a netCDF file into a UniScript object.
  Open and Close
nc_abort nc_abort closes or deletes a data file.
nc_close nc_close closes an open nc file.
nc_create nc_create creates a new netCDF file.
nc_open nc_open opens an existing data file.
nc_sync nc_sync writes all buffered data to the file.
  Global Functions
nc_copy nc_copy copies all variables and attributes of an NC file to a new NC file.
nc_diminq_size nc_diminq_size returns the size of a dimension when given its ID.
nc_endef nc_endef takes an open nc file out of define mode. The changes made to the nc file while it was in define mode are checked and committed to disk if no problems occurred. In particular, non-record variables are filled with their fill-values unless nc_setfill has been called with the argument NC_NOFILL. The nc file is then placed in data mode, so that variable data can be read or written.
nc_fatal_error nc_fatal_error.
nc_filename nc_filename returns the file name for a given ncid.
nc_get_enum_values nc_get_enum_values return a matrix with enum values or an empty string.
nc_get_option nc_get_option gets an option.
nc_inquire_format nc_inquire_format returns the file format for a given ncid.
nc_inquire_mode nc_inquire_mode returns the mode of an open nc-file.
nc_inquire_ndims nc_inquire_ndims returns the number of dimensions in a nc file.
nc_inquire_ngatts nc_inquire_ngatts returns the number of global attributes in a nc file.
nc_inquire_nvars nc_inquire_nvars returns the number of variables in a nc file.
nc_inquire_recdim nc_inquire_recdim returns the ID number of the record variable in a nc file.
nc_last_error nc_last_error returns the last error that occurred of UniPlot data files. If the last invoked nc_-function returned without an error the function will return 0.
nc_redef nc_redef puts an open netCDF file into define mode, so dimensions, variables, and attributes can be added or renamed and attributes can be deleted.
nc_set_option nc_set_option sets an option.
nc_seterror_options nc_seterror_options sets the error options for the netCDF functions.
nc_setfill nc_setfill determines whether or not variable prefilling will be done.
nc_str_error nc_str_error.
  Dimensions
nc_dimdef nc_dimdef adds a new dimension to an open netCDF file.
nc_dimid nc_dimid returns the ID of a netCDF dimension when given the name of the dimension.
nc_diminq_name nc_diminq_name returns the name of a dimension when given its ID.
nc_dimredim nc_dimredim modifies the size of a dimension. The NC2 file must be set to define mode (see nc_endef/nc_redef).
nc_dimrename nc_dimrename renames a given dimension.
  Variables
nc_makevalidname nc_makevalidname returns a valid netCDF variable name for a given name.
nc_varcopy nc_varcopy copies the data of a variable from one NC file to another NC file. The variable must exist in both files and must have the same number of points.
nc_vardef nc_vardef adds a new variable to an open netCDF file in define mode.
nc_vardelete nc_vardelete deletes a given variable. A variable can only be deleted if the file is not in definition mode.
nc_varget nc_varget reads data values from a netCDF variable of an open netCDF file. The file must be in data mode.
nc_varget_missing nc_varget_missing reads data values from a netCDF variable of an open netCDF file. The file must be in data mode.
nc_varid nc_varid returns the ID of a netCDF variable when given its name.
nc_varinq_changed nc_varinq_changed returns the change counter of a netCDF variable.
nc_varinq_datatype nc_varinq_datatype returns the data type of a variable when given its ID.
nc_varinq_dimids nc_varinq_dimids returns a vector of dimension IDs.
nc_varinq_info nc_varinq_info returns information about a channel. (Minimum, Maximum, Monotonic, channel contains missing values).
nc_varinq_name The nc_varinq_name function returns the name of a netCDF variable when given its ID.
nc_varinq_natts The nc_varinq_natts function returns the number of attributes for a variable, given its ID.
nc_varput The nc_varput functions writes data values into a netCDF variable of an open netCDF file. The file must be in data mode.
nc_varput_missing The nc_varput_missing functions writes data values into a netCDF variable of an open netCDF file. The file must be in data mode. Missing values must have the value MISSING_VALUE.
nc_varput_text The nc_varput_text functions writes string data values into a netCDF variable of an open netCDF file. The file must be in data mode.
nc_varrename nc_varrename renames a given netCDF variable.
nc_varsearch nc_varsearch searches in a given variable for the index of that value that is greater of equal than x. The values in variable must be sorted monotonically increasing and should not contain missing values.
  Attributes
nc_attcopy nc_attcopy copies a given attribute into a different NC file.
nc_attdelete nc_attdelete deletes a given attribut. The file must be in define mode.
nc_attget nc_attget returns the value(s) of a netCDF attribute.
nc_attinq_datatype nc_attinq_datatype returns the data type of an attribute when given its ID and name.
nc_attinq_len nc_attinq_len returns the number of values stored in the attribute. If the attribute is of type NC_CHAR, the function returns the string length.
nc_attname nc_attname returns the attribute name.
nc_attput nc_attput changes a variable attribute or global attribute of an open nc file. If the attribute is new, or if the space required to store the attribute is greater than before, the nc file must be in define mode.
nc_attrename nc_attrename renames a given attribute.

7.5. Standard Attributes

The netCDF file format is UniPlot’s standard data file format. These files can contain attributes of the form

AttributName = AttributValue

A netCDF attribute contains information about a netCDF variable or about the entire netCDF file. Attributes are used to specify such properties as units, special values and parameters. For example, “units” is an attribute represented by a string such as “Nm”

The attribute names are case sensitive. Units and units are different attributes.

Here is a list of attribute names that have a special meaning for UniPlot:

7.5.1. Global Attributes

Attribute Name Data Type Meaning
Origin char Name of a source file, e.g. “c:test.xls”.
Source char Import filter creator.
Creator char Name of the import filter. The name may contain a version number, e.g. UniPlot Pasy 1.0.
Date char Date when the data file was created.
Time char Time when the data file was created.
Range 2*int  
_nc_dl_driver, _nc_dl_driverinfo, _nc_dl_source, _nc_dl_loadcount char and int see Delayed Loading.

7.5.2. Channel Attributes

Attribute Name Data Type Meaning
title char A succinct description.
long_name char A long descriptive name containing the unit.
units char A string that specifies the units of the variable’s data.
scale_factor double If present, it is used to be multiplied by this factor after the data is read.
add_offset double If present, it is used to be added to the data after the data is read.
missing_value same as the channel The data type of the attribute missing_value is identical to the channel’s data type.
Type char Values is “Time Channel” or “Data Channel”
datatype char Values are: “date”, “time”, “datetime”
C_format char Format string that can be used to print values, see printf.
__Delete__ int Channel is marked as deleted if the value is 1.
Description char A description.
Comment char A comment.
_FillValue    
_formula_text char  
_formula_description char  
_formula_comment char  
_formula_onlyif char  
_channel_type char “formula” or “formula_not_registered”.
valid_range same as the channel valid_range is a channel attribute that holds two numbers specifying the minimum and maximum valid values. The type of each valid_range attribute should match the type of its variable. If a channel value is outside the valid range, it will be displayed in the grid with the font color red.
_nc_enum char The attribut contains value/text pairs. The elements are separated by an Or character (|). Example: 0|Off|1|On|2|Error. The order is Value|Text|Value|Text|.... The decimal separator is a point (.). See also nc_get_enum_values. If the attribut is available, it will be used as user labels for the y axis.
XStart_XDelta double[2] Start value and delta. Used, if a channel is used for a t/y dataset.
nc_min, nc_max, nc_monotone, nc_has_missings, nc_change_counter double See nc_varinq_info.
_nc_dl_loaded, _nc_dl_loadinfo int see Delayed Loading.

7.6. Channel Grouping (Tree Structure)

To display a channel in a group the attribute _nc_group (data type NC_CHAR) must be added to the channel, e.g. _nc_group = "Angle Data/Heat Release" or _nc_group = "G1". To enable the groups create a global attribute _nc_hasgroups = 1 (NC_LONG):

nc_attput(ncid, -1, "_nc_hasgroups", NC_BYTE, 1);
nc_attput(ncid, varid, "_nc_group", NC_CHAR, "G1");

The grouping will be displayed it the tree structur button is pressed. The Browser_ShowTreeStructure() function toggles this button. The return value 1 means the function is enabled. The global variable _g().nc_use_groups saves the setting.

7.7. Delayed Loading

If the “Delayed Channel Import” is enabled, (see Tools=>More Options, Group: Data Import and Data Browser) the data channels will be loaded from the source file into the UniPlot data file when a channel is first accessed. Example: If a file contains 400 channels and only 8 channels are loaded into an IPW document the import is 50 times faster.

Global Attributes

_nc_dl_driver = "rs_usdata"
_nc_dl_driverinfo = "TDM"
_nc_dl_source = "d:\\test.tdm"
_nc_dl_loadcount = 0

Channel-Attributes

_nc_dl_loaded = 0
_nc_dl_loadinfo = "5,2,amp"
context = _XXX_DL_open(ncid, ssSource)
bool = _XXX_DL_load_channel(context, varid, ssInfo)
_XXX_DL_close(context)

Example:

def _TDM_DL_open(ncid, ssSource)
{
    ctx = [.];
    hTDMFile = _DDC_OpenFileEx(ssSource, "", TRUE /* READONLY */);
    if (type(hTDMFile) != "error") {
        ctx.hTDMFile = hTDMFile;
        ctx.ncid = ncid;
        ctx.ssOurce = ssSource;
        return ctx;
    }
    return 0;
}
def _TDM_DL_load_channel(ctx, varid, ssInfo)
{
    sv = strtok(ssInfo, ",");
    iGroup = strtol(sv[1]);
    iChannel = strtol(sv[2]);
    oGroups = _DDC_GetChannelGroups(ctx.hTDMFile);
    oChannels = _DDC_GetChannels(oGroups[iGroup]);
    nBlocksize = 10000;
    nStart = 0;
    nRead = nBlocksize;
    while (nRead >= nBlocksize) {
        rvData = _DDC_GetDataValues(oChannels[iChannel], nStart, nBlocksize);
        if (type(rvData) == "error") {
            return FALSE;
        }
        nRead = len(rvData);
        r = nc_varput(ctx.ncid, varid, nStart, nRead, rvData);
        nStart = nStart + nRead;
    }
    return TRUE;
}
def _TDM_DL_close(ctx)
{
    _DDC_CloseFile(ctx.hTDMFile);
}

7.8. Example

The following function creates a netCDF file. In this example the data is passed as a string matrix (smData).

def NC_WriteSimpleNCFile(ssFileName, ssNetCDFName, svChanName, svUnits, ...
            rvDataType, smData, ssCreator)
{
    nCols = nc(smData);
    nRows = nr(smData);
    ncid = nc_create(ssNetCDFName);
    if (ncid == -1) {
        ssError = sprintf(_s("Cannot create file %s - Is file already open?");
        MessageBox(ssError, ssNetCDFName), "ICONSTOP");
        return "#IMPORTERROR#";
    }
    dimids = nc_dimdef(ncid, "nRows", nRows);
    // Globale Attribute:
    nc_attput(ncid, -1, "Origin", NC_CHAR, _StandardFileName(ssFileName));
    nc_attput(ncid, -1, "Source", NC_CHAR, "FEV Software and Testing Solutions GmbH");
    nc_attput(ncid, -1, "Creator", NC_CHAR, ssCreator);
    rvRange = [1, nRows];
    nc_attput(ncid, -1, "Range", NC_LONG, rvRange);
    nc_attput(ncid, -1, "Reference", NC_CHAR, "");
    nc_attput(ncid, -1, "Date", NC_CHAR, sum(date()));
    nc_attput(ncid, -1, "Time", NC_CHAR, sum(time()));
    nc_attput(ncid, -1, "Title", NC_CHAR, "");
    nc_attput(ncid, -1, "Comment1", NC_CHAR, "");
    nc_attput(ncid, -1, "Comment2", NC_CHAR, "");
    nc_attput(ncid, -1, "Comment3", NC_CHAR, "");
    for (i in 1:nCols) {
        if (NC_DATETIME == rvDataType[i]) {
            varid = nc_vardef(ncid, svChanName[i], NC_DOUBLE, dimids);
        } else {
            varid = nc_vardef(ncid, svChanName[i], rvDataType[i], dimids);
        }
        nc_attput(ncid, varid, "title", NC_CHAR, svChanName[i]);
        nc_attput(ncid, varid, "units", NC_CHAR, svUnits[i]);
        nc_attput(ncid, varid, "long_name", NC_CHAR, svChanName[i] + ..
                               "  [" + svUnits[i] + "]");
        nc_attput(ncid, varid, "scale_factor", NC_DOUBLE, 1.0);
        nc_attput(ncid, varid, "add_offset", NC_DOUBLE, 0.0);
        nc_attput(ncid, varid, "Description", NC_CHAR, "");
        nc_attput(ncid, varid, "Comment", NC_CHAR, "");
        nc_attput(ncid, varid, "ChanType", NC_CHAR, "DateTime");
    }
    nc_setfill(ncid, NC_NOFILL);
    nc_endef(ncid);
    for (i in 1:nCols) {
        if (rvDataType[i] == 100) {
            rvData = DT_ParseDateTime(smData[;i]);
        } else {
            rvData = strtod(smData[;i]); // string to double
        }
        nc_varput(ncid, i-1, 0, nRows, rvData);
    }
    nc_close(ncid);
    return ssNetCDFName;
}