Read a Teledyne/RDI ADCP file (called 'adp' in oce).

read.adp.rdi(
  file,
  from,
  to,
  by,
  tz = getOption("oceTz"),
  longitude = NA,
  latitude = NA,
  type = c("workhorse"),
  monitor = FALSE,
  despike = FALSE,
  processingLog,
  testing = FALSE,
  debug = getOption("oceDebug"),
  ...
)

Arguments

file

a connection or a character string giving the name of the file to load. (For read.adp.sontek.serial, this is generally a list of files, which will be concatenated.)

from

indication of the first profile to read. This can be an integer, the sequence number of the first profile to read, or a POSIXt time before which profiles should be skipped, or a character string that converts to a POSIXt time (assuming UTC timezone). See “Examples”, and make careful note of the use of the tz argument. If from is not supplied, it defaults to 1.

to

an optional indication of the last profile to read, in a format as described for from. As a special case, to=0 means to read the file to the end. If to is not supplied, then it defaults to 0.

by

an optional indication of the stride length to use while walking through the file. If this is an integer, then by-1 profiles are skipped between each pair of profiles that is read, e.g. the default by=1 means to read all the data. (For RDI files only, there are some extra features to avoid running out of memory; see “Memory considerations”.)

tz

character string indicating time zone to be assumed in the data.

longitude

optional signed number indicating the longitude in degrees East.

latitude

optional signed number indicating the latitude in degrees North.

type

character string indicating the type of instrument.

monitor

boolean, set to TRUE to provide an indication of progress in reading the file, either by printing a dot for each profile or by writing a textual progress bar with txtProgressBar().

despike

if TRUE, despike() will be used to clean anomalous spikes in heading, etc.

processingLog

if provided, the action item to be stored in the log. (Typically only provided for internal calls; the default that it provides is better for normal calls by a user.)

testing

logical value (IGNORED).

debug

a flag that turns on debugging. Set to 1 to get a moderate amount of debugging information, or to 2 to get more.

...

optional additional arguments that some (but not all) read.adp.*() functions pass to lower-level functions.

Value

An adp object. The contents of that object make sense for the particular instrument type under study, e.g. if the data file contains NMEA strings, then navigational data will be stored in an item called nmea in the data slot).

Details

As of 2016-09-25, this function has provisional functionality to read data from the new "SentinelV" series ADCP -- essentially a combination of a 4 beam workhorse with an additional vertical centre beam.

If a heading bias had been set with the EB command during the setup for the deployment, then a heading bias will have been stored in the file's header. This value is stored in the object's metadata as metadata$heading.bias. Importantly, this value is subtracted from the headings stored in the file, and the result of this subtraction is stored in the objects heading value (in data$heading). It should be noted that read.adp.rdi() was tested for firmware version 16.30. For other versions, there may be problems. For example, the serial number is not recognized properly for version 16.28.

In Teledyne/RDI ADP data files, velocities are coded to signed 2-byte integers, with a scale factor being used to convert to velocity in metres per second. These two facts control the maximum recordable velocity and the velocity resolution, values that may be retrieved for an ADP object name d with d[["velocityMaximum"]] and d[["velocityResolution"]].

Names of items in data slot

The names of items in the data slot are below. Not all items are present for ll file varieties; use e.g. names(d[["data"]]) to determine the names used in an object named d. In this list, items are either a vector (with one sample per time of measurement), a matrix with first index for time and second for bin number, or an array with first index for time, second for bin number, and third for beam number. Items are of vector type, unless otherwise indicated.

ItemMeaning
asignal amplitude array (units?)
ambientTempambient temperature (degC)
attitudeattitude (deg)
attitudeTemp(FIXME add a description here)
avgMagnitudeVelocityEast(FIXME add a description here)
avgMagnitudeVelocityNorth(FIXME add a description here)
avgSpeed(FIXME add a description here)
avgTrackMagnetic(FIXME add a description here)
avgTrackTrue(FIXME add a description here)
avgTrueVelocityEast(FIXME add a description here)
avgTrueVelocityNorth(FIXME add a description here)
brbottom range matrix (m)
bvbottom velocity matrix (m/s)
contaminationSensor(FIXME add a description here)
depthdepth (m)
directionMadeGood(FIXME add a description here)
distance(FIXME add a description here)
firstLatitudelatitude at start of profile (deg)
firstLongitudelongitude at start of profile (deg)
firstTime(FIXME add a description here)
gdata goodness matrix (units?)
headinginstrument heading (degrees)
headingStdinstrument heading std-dev (deg)
lastLatitudelatitude at end of profile (deg)
lastLongitudelongitude at end of profile (deg)
lastTime(FIXME add a description here)
numberOfHeadingSamplesAveraged(FIXME add a description here)
numberOfMagneticTrackSamplesAveraged(FIXME add a description here)
numberOfPitchRollSamplesAveraged(FIXME add a description here)
numberOfSpeedSamplesAveraged(FIXME add a description here)
numberOfTrueTrackSamplesAveraged(FIXME add a description here)
pitchinstrument pitch (deg)
pitchStdinstrument pitch std-dev (deg)
pressurepressure (dbar)
pressureMinus(FIXME add a description here)
pressurePlus(FIXME add a description here)
pressureStdpressure std-dev (dbar)
primaryFlags(FIXME add a description here)
qdata quality array
rollinstrument roll (deg)
rollStdinstrument roll std-dev (deg)
salinitysalinity
shipHeadingship heading (deg)
shipPitchship pitch (deg)
shipRollship roll (deg)
soundSpeedsound speed (m/s)
speedMadeGoodspeed over ground (?) (m/s)
speedMadeGoodEast(FIXME add a description here)
speedMadeGoodNorth(FIXME add a description here)
temperaturetemperature (degC)
timeprofile time (POSIXct)
vvelocity array (m/s)
xmitCurrenttransmit current (unit?)
xmitVoltagetransmit voltage

Memory considerations

For RDI files only, and only in the case where by is not specified, an attempt is made to avoid running out of memory by skipping some profiles in large input files. This only applies if from and to are both integers; if they are times, none of the rest of this section applies.

A key issue is that RDI files store velocities in 2-byte values, which is not a format that R supports. These velocities become 8-byte (numeric) values in R. Thus, the R object created by read.adp.rdi will require more memory than that of the data file. A scale factor can be estimated by ignoring vector quantities (e.g. time, which has just one value per profile) and concentrating on matrix properties such as velocity, backscatter, and correlation. These three elements have equal dimensions. Thus, each 4-byte slide in the data file (2 bytes + 1 byte + 1 byte) corresponds to 10 bytes in the object (8 bytes + 1 byte + 1 byte). Rounding up the resultant 10/4 to 3 for safety, we conclude that any limit on the size of the R object corresponds to a 3X smaller limit on file size.

Various things can limit the size of objects in R, but a strong upper limit is set by the space the operating system provides to R. The least-performant machines in typical use appear to be Microsoft-Windows systems, which limit R objects to about 2e6 bytes (see ?Memory-limits). Since R routinely duplicates objects for certain tasks (e.g. for call-by-value in function evaluation), read.adp.rdi uses a safety factor in its calculation of when to auto-decimate a file. This factor is set to 3, based partly on the developers' experience with datasets in their possession. Multiplied by the previously stated safety factor of 3, this suggests that the 2 GB limit on R objects corresponds to approximately a 222 MB limit on file size. In the present version of read.adp.rdi, this value is lowered to 200 MB for simplicity. Larger files are considered to be "big", and are decimated unless the user supplies a value for the by argument.

The decimation procedure has two cases.

  1. If from=1 and to=0 (or if neither from or to is given), then the intention is to process the full span of the data. If the input file is under 200 MB, then by defaults to 1, so that all profiles are read. For larger files, by is set to the ceiling() of the ratio of input file size to 200 MB.

  2. If from exceeds 1, and/or to is nonzero, then the intention is to process only an interior subset of the file. In this case, by is calculated as the ceiling() of the ratio of bbp*(1+to-from) to 200 MB, where bbp is the number of file bytes per profile. Of course, by is set to 1, if this ratio is less than 1.

If the result of these calculations is that by exceeds 1, then messages are printed to alert the user that the file will be decimated, and also monitor is set to TRUE, so that a textual progress bar is shown.

Development Notes

An important part of the work of this function is to recognize what will be called "data chunks" by two-byte ID sequences. This function is developed in a practical way, with emphasis being focussed on data files in the possession of the developers. Since Teledyne-RDI tends to introduce new ID codes with new instruments, that means that read.adp.rdi may not work on recently-developed instruments.

The following two-byte ID codes are recognized by read.adp.rdi at this time (with bytes listed in natural order, LSB byte before MSB). Items preceded by an asterisk are recognized, but not handled, and so produce a warning.

Byte 1Byte 2Meaning
0x000x01velocity
0x000x01velocity
0x000x02correlation
0x000x03echo intensity
0x000x04percent good
0x000x06bottom track
0x000x0aSentinel vertical beam velocity
0x000x0bSentinel vertical beam correlation
0x000x0cSentinel vertical beam amplitude
0x000x0dSentinel vertical beam percent good
0x000x20VMDASS
\*0x000x30binary fixed attitude (see p169 of reference 3 for format)
0x000x32Sentinel transformation matrix
0x000x0aSentinel data
0x000x0bSentinel correlation
0x000x0cSentinel amplitude
0x000x0dSentinel percent good
0x010x0f?? something to do with V series and 4-beam

Lacking a comprehensive Teledyne-RDI listing of ID codes, the authors have cobbled together a listing from documents to which they have access, as follows.

  • Table 33 of reference 3 lists codes as follows:

    Standard IDStandard plus 1DDESCRIPTION
    MSB LSBMSB LSB
    --- ------ ---
    7F 7F7F 7FHeader
    00 0000 01Fixed Leader
    00 8000 81Variable Leader
    01 0001 01Velocity Profile Data
    02 0002 01Correlation Profile Data
    03 0003 01Echo Intensity Profile Data
    04 0004 01Percent Good Profile Data
    05 0005 01Status Profile Data
    06 0006 01Bottom Track Data
    20 0020 00Navigation
    30 0030 00Binary Fixed Attitude
    30 40-F030 40-F0Binary Variable Attitude
  • Table 6 on p90 of reference 4 lists "Fixed Leader Navigation" ID codes (none of which are handled by read.adp.rdi yet) as follows (the format is reproduced literally; note that e.g. 0x2100 is 0x00,0x21 in the oce notation):

    IDDescription
    0x2100$xxDBT
    0x2101$xxGGA
    0x2102$xxVTG
    0x2103$xxGSA
    0x2104$xxHDT, $xxHGD or $PRDID

    and following pages in that manual reveal the following meanings

    SymbolMeaning
    DBTdepth below transducer
    GGAglobal positioning system
    VTAtrack made good and ground speed
    GSAGPS DOP and active satellites
    HDTheading, true
    HDGheading, deviation, and variation
    PRDIDheading, pitch and roll

Error recovery

Files can sometimes be corrupted, and read.adp.rdi has ways to handle two types of error that have been noticed in files supplied by users.

  1. There are two bytes within each ensemble that indicate the number of bytes to check within that ensemble, to get the checksum. Sometimes, those two bytes can be erroneous, so that the wrong number of bytes are checked, leading to a failed checksum. As a preventative measure, read.adp.rdi checks the stated ensemble length, whenever it detects a failed checksum. If that length agrees with the length of the most recent ensemble that had a good checksum, then the ensemble is declared as faulty and is ignored. However, if the length differs from that of the most recent accepted ensemble, then read.adp.rdi goes back to just after the start of the ensemble, and searches forward for the next two-byte pair, namely 0x7f 0x7f, that designates the start of an ensemble. Distinct notifications are given about these two cases, and they give the byte numbers in the original file, as a way to help analysts who want to look at the data stream with other tools.

  2. At the end of an ensemble, the next two characters ought to be 0x7f 0x7f, and if they are not, then the next ensemble is faulty. If this error occurs, read.adp.rdi attempts to recover by searching forward to the next instance of this two-byte pair, discarding any information that is present in the mangled ensemble.

In each of these cases, warnings are printed about ensembles that seem problematic. Advanced users who want to diagnose the problem further might find it helpful to examine the original data file using other tools. To this end, read.adp.rdi inserts an element named ensembleInFile into the metadata slot. This gives the starting byte number of each inferred ensemble within the original data file. For example, if d is an object read with read.adp.rdi, then using

plot(d[["time"]][-1], diff(d[["ensembleInFile"]]))

can be a good way to narrow in on problems.

References

  1. Teledyne-RDI, 2007. WorkHorse commands and output data format. P/N 957-6156-00 (November 2007). (Section 5.3 h details the binary format, e.g. the file should start with the byte 0x7f repeated twice, and each profile starts with the bytes 0x80, followed by 0x00, followed by the sequence number of the profile, represented as a little-endian two-byte short integer. read.adp.rdi uses these sequences to interpret data files.)

  2. Teledyne RD Instruments, 2015. V Series monitor, sentinel Output Data Format. P/N 95D-6022-00 (May 2015). SV_ODF_May15.pdf

  3. Teledyne RD Instruments, 2014. Ocean Surveyor / Ocean Observer Technical Manual. P/N 95A-6012-00 (April 2014). OS_TM_Apr14.pdf

  4. Teledyne RD Instruments, 2001. WinRiver User's Guide International Version. P/N 957-6171-00 (June 2001) WinRiver User Guide International Version.pdf.pdf

See also

Examples

adp <- read.adp.rdi(system.file("extdata", "adp_rdi.000", package="oce")) summary(adp)
#> ADP Summary #> ----------- #> #> * Instrument: adcp #> * Serial number: unknown #> * Firmware version: 16.28 #> * Source filename: ``/private/var/folders/rx/k1x_t_q12zb9hbjr4ldxmjc80000gn/T/RtmpjZN348/temp_libpath17c45663e6dd0/oce/extdata/adp_rdi.000`` #> * Location: unknown latitude, unknown longitude #> * Number of profiles: 9 #> * Number of cells: 84 #> * Number of beams: 4 #> * Cell size: 0.5 m #> * Summary of times between profiles: #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 10 10 10 10 10 10 #> * Frequency: 600 kHz #> * Ensemble Numbers: object@metadata$ensembleNumber [1:9]: 1, 2, ..., 8, 9 #> * Cells: 84, centered at 2.230 m to 43.730 m, spaced by 0.500 m #> * Coordinate system: beam [originally], beam [presently] #> * Beams:: #> Number: 4 #> Slantwise Angle: 20 #> Orientation: upward:9 #> Unspreaded: FALSE #> * Transformation matrix:: #> 1.462 -1.462 0.000 0.000 #> 0.000 0.000 -1.462 1.462 #> 0.266 0.266 0.266 0.266 #> 1.034 1.034 -1.034 -1.034 #> * Time ranges from 2008-06-25 10:00:00 to 2008-06-25 10:01:20 with 9 samples and mean increment 10 s #> * Data Overview #> #> Min. Mean Max. Dim. NAs OriginalName #> v [m/s] -0.416 0.011534 0.406 9x84x4 0 - #> q 16 25.18 34 9x84x4 0 - #> a 45 50.164 55 9x84x4 0 - #> g 100 100 100 9x84x4 0 - #> distance [m] 2.23 22.98 43.73 84 0 - #> pressure [dbar] -0.274 -0.23489 -0.193 9 0 - #> temperature [°C, ITS-90] 12.06 12.082 12.11 9 0 - #> salinity [PSS-78] 35 35 35 9 0 - #> depth [m] 0 0 0 9 0 - #> soundSpeed [m/s] 1497 1497 1497 9 0 - #> heading [°] 276.39 277.14 278.14 9 0 - #> pitch [°] 1.1209 1.2033 1.4212 9 0 - #> roll [°] -2.49 -2.4033 -2.35 9 0 - #> headingStd [°] 0 0.22222 1 9 0 - #> pitchStd [°] 0.1 0.11111 0.2 9 0 - #> rollStd [°] 0 0.088889 0.1 9 0 - #> pressureStd 76 100 135 9 0 - #> xmitCurrent 61 61 61 9 0 - #> xmitVoltage 155 155 155 9 0 - #> ambientTemp 103 103 103 9 0 - #> pressurePlus 77 77 77 9 0 - #> pressureMinus 76 76.778 77 9 0 - #> attitudeTemp 101 101 101 9 0 - #> attitude [°] 130 130 130 9 0 - #> contaminationSensor 159 159 159 9 0 - #> #> * Processing Log #> #> - 2020-07-21 16:51:53 UTC: `read.adp.rdi(file = system.file("extdata", "adp_rdi.000", package = "oce"))`