.. highlightlang:: us .. index:: Overview netCDF Files, netCDF Files .. _overview-netcdf-files: Overview UniPlot Data Files =========================== If you import data into UniPlot, the data will be saved in an UniPlot data file format (nc2). Up until UniPlot version R2011, UniPlot could create netCDF files. The standard extension is :file:`.nc`. netCDF is a file format for measured data developed by the University Corporation for Atmospheric Research (see :ref:`copyright-netcdf`). Beginning with UniPlot R2012, netCDF files are not used to load data. Content of a UniPlot Data File (nc2) ------------------------------------ The following listing contains the output of :ref:`ncdump_exe`. The program lists the contents of a nc- or nc2 file as a text file. ( :ref:`ncgen_exe` can be used to convert the text file into a nc2 file.) .. code-block:: none :linenos: netcdf Test1.xls { dimensions: n = 5 ; variables: float EngSpd(n) ; EngSpd:title = "EngSpd" ; EngSpd:long_name = "EngSpd [RPM]" ; EngSpd:units = "RPM" ; float PME(n) ; PME:title = "PME" ; PME:long_name = "PME [bar]" ; PME:units = "bar" ; // global attributes: :Origin = "D:\\uniplot\\samples\\Test1.xls" ; :Source = "FEV Software and Testing Solutions GmbH (www.uniplot.de)" ; :Creator = "UniPlot Excel Converter v3" ; data: EngSpd = 1006.00, 1249.00, 1512.00, 1708.00, 1804.00; PME = 8.47, 9.33, 10.64, 11.21, 11.27; } .. highlightlang:: us Dimensions ^^^^^^^^^^^ Each channel has a named dimension. All channels with the same dimension are displayed together in a table in the :ref:`Data Editor. ` If the channel ``A`` has the dimension ``n1 = 10`` (10 points) and the channel ``B`` has the dimension ``n2 = 10`` (also 10 points), the channels will be displayed in the data editor in two two different tables. A channel can also contain several dimensions, but UniPlot uses this property only for string channels. The first dimension is the number of strings and the second dimension is the length of the longest string. String channels are a matrix of characters. Attributes ^^^^^^^^^^ An attribute is a key/value pair. The key is a name and the value element is a scalar number, or a vector of numbers, or a string. UniPlot data files supports the following data types for channels and attributes: .. list-table:: :header-rows: 1 * - Data Type - Meaning * - NC_CHAR - Character (8 bit). * - NC_INT8 - Integer numbers with 8 bits (1 Byte). (Range -127 to 128 or 0 to 255.) * - NC_INT16 - Integer numbers with 16 bits (2 Bytes). * - NC_INT32 - Integer numbers with 32 bits (4 Bytes). * - NC_FLOAT - Real number with 32 bits (4 Bytes). * - NC_DOUBLE - Real numbers with 64 bits (8 Bytes). Besides the data type, an attribute has a length. If the data type is NC_CHAR, the length is the number of bytes. For all other types the length specifies the number of values in the attribute. The attribute can only be accessed as a complete vector. For instance, to modify a character in a string, the complete string must be read, modified and written back. The size of an attribute should be small (less than 1,000 bytes). A UniPlot data file can contain up to 20,000 global attributes. Global is the name of the attributes because they refer to the whole file. The channels also contain a list of attributes, the channel attributes. An UniPlot data file can contain an unlimited number of attributes. A file can contain channel attributes and global attributes. See :ref:`netcdf-attributes` for more about writing attributes. Channel ^^^^^^^ Channels are also Key/Value pairs. The key is the channel name and the value is a vector of numbers. The vector can contain up to 2.1 billion elements. :: time = [0.1, 0.2, 0.3, ..., 2.000.000] Channel Names ------------- Channel names may contain the following special characters: ``.``, ``-``, ``+``, ``$``, ``#``, ``~``, ``!``, ``^``, ``&``, ``%``. To avoid problems with the formula interpreter, data exchange or compatibility problems, the channel name can be used with underscores instead of the special characters. Example: A NC-file contains a the channel ``AI50%+m``. The following two calls will return the correct, identical *varid*:: varid = nc_varid(ncid, "AI50%+m"); varid = nc_varid(ncid, "AI50__m"); If the NC file contains channel names which only differ in special characters the function may return different varids. The reason is that the function returns the first channel id that matches the channel name. Example: A NC file contains the following two channels ``AI50%+m`` (varid 0) and ``AI50%-m`` (varid 1): The following call will return the *varid* = 1:: varid = nc_varid(ncid, "AI50%-m"); If called with two underscores the function returns the *varid* of ``AI50%+m``, *varid* = 0:: varid = nc_varid(ncid, "AI50__m"); To avoid problems, all channel names should be unique even if the special characters are replaced by underscores. If the NC files are created with UniPlot, the channel names which are only different in the special characters are enumerated, starting with 0. The first channel name will be unmodified. .. _netcdf-up: Properties of UniPlot data files (nc2) -------------------------------------- * Usually, when editing a data file (adding channels, remove channels) the results for most data formats is writing a modified copy of the file. This is not necessary for nc2 files. This enables much higher performances for these operations. * Stability: In netCDF :ref:`nc_abort` cannot be used to undo modifications. If a number of changes are executed and one modification fails because of power failure, a network problem or a software bug the complete file is destroyed. For nc2 files all modifications can be undone because the files use transactions. * Reduced file size, because parts of the file are saved compressed. The following table compares the file size of three typical MDF/INCA files with netCDF- and netCDF-up files (all values in kBytes): .. list-table:: :header-rows: 1 * - File - Original (MDF/VS100) - netCDF - nc2 * - File small - 11 - 17 - 12 * - File medium - 934 - 1426 - 453 * - File big - 13727 - 33883 - 4271 Sometimes (for example pure random numbers) nc2 files can be bigger than netCDF fields. This will be a rare case. * No Limitation in size - no 2 GBytes limit. File size is limited to 1 TB. Channel size is limited to 2.1 billion data points. .. _functions: Functions --------- .. us.makeindex nc.tools, Tools .. include:: ../ftab/nc-tools.ftab .. us.makeindex NC_open, Open and Close .. include:: ../ftab/NC_open.ftab .. us.makeindex NC, Global Functions .. include:: ../ftab/NC.ftab .. us.makeindex NC_Dim, Dimensions .. include:: ../ftab/NC_Dim.ftab .. us.makeindex NC_Var, Variables .. include:: ../ftab/NC_Var.ftab .. us.makeindex NC_Att, Attributes .. include:: ../ftab/NC_Att.ftab .. _netcdf-attributes: Standard Attributes ------------------- The netCDF file format is UniPlot's standard data file format. These files can contain attributes of the form :: AttributName = AttributValue A netCDF attribute contains information about a netCDF variable or about the entire netCDF file. Attributes are used to specify such properties as units, special values and parameters. For example, "units" is an attribute represented by a string such as "Nm" The attribute names are case sensitive. ``Units`` and ``units`` are different attributes. Here is a list of attribute names that have a special meaning for UniPlot: .. _global-attributes: Global Attributes ^^^^^^^^^^^^^^^^^ .. list-table:: :header-rows: 1 * - Attribute Name - Data Type - Meaning * - Origin - char - Name of a source file, e.g. "c:\test.xls". * - Source - char - Import filter creator. * - Creator - char - Name of the import filter. The name may contain a version number, e.g. UniPlot Pasy 1.0. * - Date - char - Date when the data file was created. * - Time - char - Time when the data file was created. * - Range - 2\*int - * - _nc_dl_driver, _nc_dl_driverinfo, _nc_dl_source, _nc_dl_loadcount - char and int - see :ref:`delay-loading`. .. _data-channel-attributes: Channel Attributes ^^^^^^^^^^^^^^^^^^ .. list-table:: :header-rows: 1 * - Attribute Name - Data Type - Meaning * - title - char - A succinct description. * - long_name - char - A long descriptive name containing the unit. * - units - char - A string that specifies the units of the variable's data. * - scale_factor - double - If present, it is used to be multiplied by this factor after the data is read. * - add_offset - double - If present, it is used to be added to the data after the data is read. * - missing_value - same as the channel - The data type of the attribute missing_value is identical to the channel's data type. * - Type - char - Values is "Time Channel" or "Data Channel" * - datatype - char - Values are: "date", "time", "datetime" * - C_format - char - Format string that can be used to print values, see :ref:`printf`. * - __Delete__ - int - Channel is marked as deleted if the value is 1. * - Description - char - A description. * - Comment - char - A comment. * - _FillValue - - * - _formula_text - char - * - _formula_description - char - * - _formula_comment - char - * - _formula_onlyif - char - * - _channel_type - char - "formula" or "formula_not_registered". * - valid_range - same as the channel - ``valid_range`` is a channel attribute that holds two numbers specifying the minimum and maximum valid values. The type of each ``valid_range`` attribute should match the type of its variable. If a channel value is outside the valid range, it will be displayed in the grid with the font color red. * - _nc_enum - char - The attribut contains value/text pairs. The elements are separated by an Or character (|). Example: ``0|Off|1|On|2|Error``. The order is ``Value|Text|Value|Text|...``. The decimal separator is a point (.). See also :ref:`nc_get_enum_values`. If the attribut is available, it will be used as user labels for the y axis. * - XStart_XDelta - double[2] - Start value and delta. Used, if a channel is used for a t/y dataset. * - nc_min, nc_max, nc_monotone, nc_has_missings, nc_change_counter - double - See :ref:`nc_varinq_info`. * - _nc_dl_loaded, _nc_dl_loadinfo - int - see :ref:`delay-loading`. Channel Grouping (Tree Structure) --------------------------------- To display a channel in a group the attribute ``_nc_group`` (data type NC_CHAR) must be added to the channel, e.g. ``_nc_group = "Angle Data/Heat Release"`` or ``_nc_group = "G1"``. To enable the groups create a global attribute ``_nc_hasgroups = 1`` (NC_LONG):: nc_attput(ncid, -1, "_nc_hasgroups", NC_BYTE, 1); nc_attput(ncid, varid, "_nc_group", NC_CHAR, "G1"); The grouping will be displayed it the tree structur button is pressed. The ``Browser_ShowTreeStructure()`` function toggles this button. The return value 1 means the function is enabled. The global variable ``_g().nc_use_groups`` saves the setting. .. _delay-loading: Delayed Loading --------------- If the "Delayed Channel Import" is enabled, (see :ref:`tools-more-options`, Group: Data Import and Data Browser) the data channels will be loaded from the source file into the UniPlot data file when a channel is first accessed. Example: If a file contains 400 channels and only 8 channels are loaded into an IPW document the import is 50 times faster. Global Attributes :: _nc_dl_driver = "rs_usdata" _nc_dl_driverinfo = "TDM" _nc_dl_source = "d:\\test.tdm" _nc_dl_loadcount = 0 Channel-Attributes :: _nc_dl_loaded = 0 _nc_dl_loadinfo = "5,2,amp" :: context = _XXX_DL_open(ncid, ssSource) bool = _XXX_DL_load_channel(context, varid, ssInfo) _XXX_DL_close(context) Example:: def _TDM_DL_open(ncid, ssSource) { ctx = [.]; hTDMFile = _DDC_OpenFileEx(ssSource, "", TRUE /* READONLY */); if (type(hTDMFile) != "error") { ctx.hTDMFile = hTDMFile; ctx.ncid = ncid; ctx.ssOurce = ssSource; return ctx; } return 0; } def _TDM_DL_load_channel(ctx, varid, ssInfo) { sv = strtok(ssInfo, ","); iGroup = strtol(sv[1]); iChannel = strtol(sv[2]); oGroups = _DDC_GetChannelGroups(ctx.hTDMFile); oChannels = _DDC_GetChannels(oGroups[iGroup]); nBlocksize = 10000; nStart = 0; nRead = nBlocksize; while (nRead >= nBlocksize) { rvData = _DDC_GetDataValues(oChannels[iChannel], nStart, nBlocksize); if (type(rvData) == "error") { return FALSE; } nRead = len(rvData); r = nc_varput(ctx.ncid, varid, nStart, nRead, rvData); nStart = nStart + nRead; } return TRUE; } def _TDM_DL_close(ctx) { _DDC_CloseFile(ctx.hTDMFile); } .. _example-nc: Example ------- The following function creates a netCDF file. In this example the data is passed as a string matrix (``smData``). :: def NC_WriteSimpleNCFile(ssFileName, ssNetCDFName, svChanName, svUnits, ... rvDataType, smData, ssCreator) { nCols = nc(smData); nRows = nr(smData); ncid = nc_create(ssNetCDFName); if (ncid == -1) { ssError = sprintf(_s("Cannot create file %s - Is file already open?"); MessageBox(ssError, ssNetCDFName), "ICONSTOP"); return "#IMPORTERROR#"; } dimids = nc_dimdef(ncid, "nRows", nRows); // Globale Attribute: nc_attput(ncid, -1, "Origin", NC_CHAR, _StandardFileName(ssFileName)); nc_attput(ncid, -1, "Source", NC_CHAR, "FEV Software and Testing Solutions GmbH"); nc_attput(ncid, -1, "Creator", NC_CHAR, ssCreator); rvRange = [1, nRows]; nc_attput(ncid, -1, "Range", NC_LONG, rvRange); nc_attput(ncid, -1, "Reference", NC_CHAR, ""); nc_attput(ncid, -1, "Date", NC_CHAR, sum(date())); nc_attput(ncid, -1, "Time", NC_CHAR, sum(time())); nc_attput(ncid, -1, "Title", NC_CHAR, ""); nc_attput(ncid, -1, "Comment1", NC_CHAR, ""); nc_attput(ncid, -1, "Comment2", NC_CHAR, ""); nc_attput(ncid, -1, "Comment3", NC_CHAR, ""); for (i in 1:nCols) { if (NC_DATETIME == rvDataType[i]) { varid = nc_vardef(ncid, svChanName[i], NC_DOUBLE, dimids); } else { varid = nc_vardef(ncid, svChanName[i], rvDataType[i], dimids); } nc_attput(ncid, varid, "title", NC_CHAR, svChanName[i]); nc_attput(ncid, varid, "units", NC_CHAR, svUnits[i]); nc_attput(ncid, varid, "long_name", NC_CHAR, svChanName[i] + .. " [" + svUnits[i] + "]"); nc_attput(ncid, varid, "scale_factor", NC_DOUBLE, 1.0); nc_attput(ncid, varid, "add_offset", NC_DOUBLE, 0.0); nc_attput(ncid, varid, "Description", NC_CHAR, ""); nc_attput(ncid, varid, "Comment", NC_CHAR, ""); nc_attput(ncid, varid, "ChanType", NC_CHAR, "DateTime"); } nc_setfill(ncid, NC_NOFILL); nc_endef(ncid); for (i in 1:nCols) { if (rvDataType[i] == 100) { rvData = DT_ParseDateTime(smData[;i]); } else { rvData = strtod(smData[;i]); // string to double } nc_varput(ncid, i-1, 0, nRows, rvData); } nc_close(ncid); return ssNetCDFName; } .. _copyright-netcdf: Copyright netCDF ---------------- See :ref:`netCDF-copyright` and http://www.unidata.ucar.edu/software/netcdf/ .. seealso:: :ref:`overview-files`, :ref:`functions-by-categories`, :ref:`overview-netcdf-file-browser`, :ref:`application-limits`, :ref:`netCDF-SDK` :sub:`id-1707929`