Area Estimation of Deep-Sea Surfaces from Oblique Still Images

Estimating the area of seabed surfaces from pictures or videos is an important problem in seafloor surveys. This task is complex to achieve with moving platforms such as submersibles, towed or remotely operated vehicles (ROV), where the recording camera is typically not static and provides an oblique view of the seafloor. A new method for obtaining seabed surface area estimates is presented here, using the classical set up of two laser devices fixed to the ROV frame projecting two parallel lines over the seabed. By combining lengths measured directly from the image containing the laser lines, the area of seabed surfaces is estimated, as well as the camera’s distance to the seabed, pan and tilt angles. The only parameters required are the distance between the parallel laser lines and the camera’s horizontal and vertical angles of view. The method was validated with a controlled in situ experiment using a deep-sea ROV, yielding an area estimate error of 1.5%. Further applications and generalizations of the method are discussed, with emphasis on deep-sea applications.


Introduction
Estimating size, area and ground slope-besides distance-from photographic records is a longlasting problem. During the last decades, the development and broad use of underwater vehicles or gear equipped with photographic and video cameras has turned this topic into a subject of major interest [1][2][3], applicable both to quantitative ecological studies [4][5][6][7], as well as to geological research and exploration efforts [8,9].
When an image is recorded with negligible optical distortion, which is generally the case in underwater visual surveys, proportions are preserved. This means that the lengths on the picture match with the corresponding angular sizes. The problem is then how to correlate angular size with "real" linear size and use the fact that angular size decreases with distance.
A straightforward approach has been to collect images using a camera that is perpendicular to the surface of interest, and employ a scaling object to obtain area measurements. This approach has the drawback of a restricted field of view and a limited applicability in underwater platforms that require a forward-facing camera for technical or navigational purposes [10,11]. A solution for estimating linear size in oblique images is to use a ruler next to the target of interest [3,4]. This, however, is applicable neither to non-stationary targets, e.g. in scenarios involving moving organisms, nor to recording of seabed features during navigation along a transect line. A major improvement to this technique, that has become the standard solution, is to use laser spots projected on the seabed or target at the time of image recording. In particular, the use of four [12] or more spots allows the estimation of surface area values [13]. Most of these methods rely on accurately knowing the height of the camera above the ground [14]. However, this parameter is not trivial to estimate for several platforms, e.g. towed cameras [15]. Typically, it is also required to keep the laser beams parallel to the optical axis of the camera [13], which is accomplished by coupling the laser devices to the camera.
Here, we present a new method for obtaining seabed area estimates in an oblique view image using two parallel laser lines that are fixed to the platform frame. Calculations and final simplifications are presented, including formulas to obtain camera distance from the seabed, and camera pan and tilt values. The results were validated with a controlled in situ experiment using a deep-sea ROV. Some applications of the method are discussed.

Materials and Methods
Within the framework of the Portuguese Continental Shelf Extension Project, the ROV "Luso" has been in operation since 2008. It has surveyed and sampled the deep Northeast Atlantic seabed ( Fig 1A). The ROV is a Deepwater Work Class, model Argus Bathysaurus XL rated for 6000 m depth (see reference [16] for technical details). It is owned and operated by EMEPC (Portuguese Task Group for the Extension of the Continental Shelf). Amongst other sensors, the ROV is equipped with a full HD video camera that allows recording footage of benthic environments from which still images are extracted. The camera is an Argus HD-SDI camera [17], comprising an Argus HD Camera Housing with a Sony FCB-H11 camera (10× optical zoom; focal distance 5.1-51mm). Fixed on the ROV frame, on both sides of the camera, there are two underwater line laser devices (Fig 1B), each generating a sharp green line (532 nm) over the seabed (Fig 1C). The two lasers are paralleled and calibrated at a set distance, generally l * 60 cm apart. In this study, the value l = 67 cm was used for the distance between the laser lines. The laser lines do not move with respect to the ROV frame. Our mathematical solution is independent of any other specifics of the setup.
The objective is to calculate the seabed area S that corresponds to the trapezoid [A'B'D'C'] on the still picture (Fig 1C), given the distance l between the parallel laser lines. In other words, given the distance between the points A and C (or B and D) on the seabed (Fig 2), the distance L between A and B (or C and D) must be computed, by measuring the distances between the corresponding projected points A' and B' (or C' and D') on the still image, thus obtaining the area S = L×l.
In the Results and Discussion section we derive a mathematical solution to the problem that only involves elementary geometry and trigonometry. The equations are written in terms of distances in pixels that can be measured from the picture with an image analysis software such as ImageJ [18].
Our results were validated in situ during a test dive of the ROV "Luso", from the Spanish R/ V "Sarmiento de Gamboa" in the MOWER campaign (September 2014), for which no specific permits were required and that did not involve endangered or protected species. The dive was performed in the Western Mediterranean near the Bay of Cadiz, Spain at a depth of 643 m.

Results and Discussion
The calculations presented in this section are based on the following assumptions: a) the seabed is horizontal; b) the laser devices are installed at both sides of the camera ( Fig 1B); c) when the camera is aligned with the lasers, the projection [QT] on the seafloor of the optical axis [00"] is at distances l 1 and l 2 from the laser lines, with l 1 + l 2 = l (see Fig 2); d) the still image has negligible optical distortion and is not cropped; e) the camera's horizontal and vertical angles of view, α H and α V , respectively, are known; f) the center of the image coincides with the optical center.
Consider the special case of a camera with an optical axis that is horizontal and parallel to the laser lines (Fig 2). The camera is located at the origin, at a height h above the seabed. The plane of the image I is always perpendicular to the optical axis  Fig 2), the Cartesian coordinates of the seabed points A, B, C and D are given by The corresponding projections A', B', C' and D' are given by the intercept of the plane I with the line connecting the origin to the respective points on the seabed (the corresponding light rays, represented as red dotted lines in Fig 2). For example, the line segment [0A] is described by (x, y, z) = t(x A , y A , z A ) with 0 t 1; for the point A', y A 0 = f applies, thus t A 0 = f / y A , which allows to express x A 0 and z A 0 in terms of the coordinates of point A. Proceeding in the same way for the points B', C' and D', it follows that > : where all y-coordinates are equal to f. Notice that the xand z-coordinates of Eq (2) also give the points' positions in the image relative to the image center 0'.
The "perspective" angles θ 1 and θ 2 (see Fig 3A) are Note that there is no need to explicitly measure the "perspective" angles θ 1 and θ 2 , although it is possible to do so with image analysis software [18]. In fact, Eq (3) shows that it suffices to determine length ratios from the still image in order to obtain the "perspective" angles (see Eq (23) for the general case in pixels).
By adding the two equalities in Eq (3) and using Eq (2), h becomes applies to the "perspective" angles in (4). Therefore, by inserting this result into (4), Eq (6) means that, given the distance l between the laser lines, the height of the camera above the seabed may be determined by the length ratio z A 0 / l 0 measured on the still image.
Since y P = y A , z P = z A , y P 0 = y A 0 , z P 0 = z A 0 , by substituting into (7) the yand z-coordinates given by Eqs (1) and (2), it follows that Analogously, for the triangles [B0D] and [B'0'D'] it results that Combining (8) with (9) yields Let W be the picture's width, corresponding to the angular size α H (see Fig 3B). Notice that α H is the camera's horizontal angle of view, which is assumed to be known, as is generally the case. Thereby Eq (11) is always valid, even in the general case where the camera is rotated in relation to the ground, given that f is the distance to the image from the point of view of the camera.
By inserting (11) into (10), L becomes Thus, assuming that the parameters l and α H are known, the length L over the seafloor can be determined by length ratios measured on the still image.
Since the images are digital, l 0 , l@ and W are expressed as pixel counts. Let N be the horizontal size of the picture measured in pixels. Assuming that each pixel always corresponds to the same horizontal length W / N, let n 0 be the number of pixels between A' and C' (Fig 3A). Analogously, let n@ be the number of pixels between B' and D' (Fig 3a). Thus, l 0 = n 0 W / N, l@ = n@W / N, Eq (12) takes the form and the area of the seabed surface corresponding to the trapezoid [A'B'D'C'] on the still image is Note that the horizontal length per pixel W / N does not need to be determined, as it does not appear in Eqs (13) and (14).

Seabed surface area estimates accounting for camera pan and tilt
ROV seafloor surveys typically involve the bulk collection of imagery using a camera that is vertically tilted by an angle λ, while simultaneously rotated by a pan angle γ. Panning or tilting the camera moves the vanishing point of the laser lines away from the center of the image. As the camera tilts towards the ground, the vanishing point moves upward relative to the image center. A positive (negative) pan corresponds to a rotation to the right (left). This is in fact the case of Fig 1C, that corresponds to a negative value of the pan angle γ, with the camera tilted towards the seabed. Presented below are only the main equations of interest for practical applications that result from generalizing the approach discussed in the previous section. The full details on the corresponding derivation can be found in S1 Text.
The coordinates of the vanishing point V' in relation to the center of the image (see Fig 4A) are given by and Eqs (15) and (16) show that the still images with the laser lines contain all the necessary information to determine the camera's pan and tilt angles. In fact, from (16) one can estimate λ and then use (15) to get γ (see Eqs (20) and (21) below). This makes it possible to determine the length L over the seafloor for the general case (Fig 4), Since l 0 = n 0 W / N and l@ = n@W / N (see Fig 4A), Eq (16) becomes where cos λ and cos λ are given below by Eqs (20) and (21), respectively. For a non-zero value of the pan angle γ the trapezoid [A'B'D'C'] now defines a parallelogram of sides l / cos γ and L on the seafloor (see Fig 4B), as opposed to a rectangle for the case γ = 0 (see Fig 2). Nonetheless, the corresponding surface area is still given by S = L×l, which in pixel quantities (see Fig 4A) yields where the effects of the camera's tilt and pan are accounted by and Eqs (20) and (21) are expressed in terms of ratios of quantities directly obtainable from the still image (see Fig 4A). The camera's vertical angle of view α V appears in (20) because of the conversion of vertical linear dimensions into pixels.

Camera distance to the seabed
The general version of Eq (4) gives the vertical distance to the seabed: Therefore, by using (20), (21) and (23) together with Eq (22), the height of the camera relative to the seafloor may be determined by pixel length ratios measured on the picture.

Estimation of uncertainties
Consider Eq (17) for the length L over the seafloor. The value of L is a function of the measured lengths l 0 and l@, the cosines of γ and λ, and the distance l between the laser lines. The uncertainties in these quantities generate an uncertainty ΔL in the value of L. The uncertainties in the camera's angles of view α H and α V are assumed to be negligible.
The maximum value of ΔL may be estimated by [19] DL ¼ @L @l 0 Dl 0 þ @L @l@ Dl@ þ @L @cos g Dcos g þ @L @cos l Dcos l þ @L @l Dl ð24Þ where Δl 0 , Δl@, Δcos γ, Δcos λ and Δl are the uncertainties in l 0 , l@, cos γ, cos λ and l, respectively. Typically, Δl * 0.5 cm is verified. Measuring lengths in pixels, e.g. l 0 = n 0 W / N, implies Δl 0 = Δl@ = W / N. Thus, using Eqs (17) and (18) together with Eq (24), the relative uncertainty in the value of L is Since cos γ and cos λ are also determined by measurements taken from the still image, the corresponding relative uncertainties are obtained by using equations analogous to (24). From Eqs (20) and (21) it follows that and Likewise, from (18) it follows that the relative uncertainty in S is and from (21) and (22) it results that the relative uncertainty in h is where the relative uncertainties in cos γ and cos λ are given by Eqs (26) and (27), respectively.

In situ validation
In situ validation of our results was performed with the ROV "Luso", at a depth of 643 m ( Fig  5). Although our mathematical solution does not depend on depth, in practice it is easier to properly resolve the laser lines in deep waters. At depths greater than~100 m it is possible to avoid adverse shallow-water optical conditions such as bright ambient lighting and light scattering by suspended particulate matter. Specifically, four identical quadrat frames were assembled, each with inner dimensions 50×50 cm and edge thickness 2.5 cm. Each square featured an inner grid of 10 cm. Thus, a 110×110 cm square object was deployed. Due to time and operational constraints, only one test dive was performed. All frames available from the existing footage are equivalent for the purposes of testing Eqs (19) and (22). Therefore, a single image was analyzed (Fig 5) and the area of the object located between the parallel laser lines was estimated. The still image was acquired with the ROV at rest on the seafloor. The height of the camera relative to the ground is h exp = 113 ± 1 cm and the laser lines are l = 67 ± 0.5 cm apart. The horizontal and vertical angles of view for the still image are α H = 50.43°and α V = 29.67°, respectively. Fig 5 shows that the test frame is slightly deformed due to the influence of the terrain conditions on the assembly geometry. This effect is accounted by considering that the linear distortion of the frame results in L exp = 108 ± 3 cm. Thus, the trapezoid [A'B'D'C'] corresponds to an area of S exp = L exp × l = 7236 cm 2 . The uncertainty in S exp is estimated from ΔS exp / S exp = ΔL exp / L exp + Δl / l, which yields ΔS exp = 255 cm 2 (i.e. a percentage uncertainty of 3.5%).
Given these measurements, Eq (18) yields L = 106.4 cm, affected by an uncertainty ΔL = 5.5 cm estimated from (25). As for the area of the seabed surface corresponding to the trapezoid [A'B'D'C'], from Eqs (19) and (28) it results that S = 71230 cm 2 with an uncertainty ΔS = 425 cm 2 . These values are in excellent agreement with the respective experimental values. In fact, L differs *1.5% from L exp and S differs *1.5% from S exp . From Eq (22) it follows that the height of the camera relative to the seafloor is h = 100.7 cm, and (29) yields Δh = 5.8 cm for the uncertainty. This is still an acceptable result, as h differs *11% from the true value h exp .

Conclusions
The common usage of laser scaling devices in underwater image platforms is to have the lasers moving with the camera. This is generally the case for spot generating laser beams. Such a configuration is advantageous when estimating the dimensions of specific conspicuous objects or macro-organisms, as the camera and the laser spots are directed towards the target. However, it does not allow obtaining wide field area estimates.
This was achieved by [13] using five laser diodes with the Automated Benthic Image-Scaling System (ABISS). The method involves aligning four laser diodes in parallel to project the corners of a square of known dimensions, and placing a fifth unaligned laser in the same plane as the bottom pair of parallel lasers. The setup allows to calculate the camera to object distance. As camera orientation with respect to the substratum changes, the parallel laser spots projected onto the seabed appear as a trapezium. Finally, the Benthic Imager software (University of Plymouth, UK) calculates the actual seabed area contained within an image [5]. However, the installation of the five lasers around the camera lens can be complex, and a trigonometry solving software is required [5,13]. Lasers are often separated by short distances, and for wide angle views or as the recording platform increases the distance from the seafloor, the laser spots tend to disappear from sight, due to the relatively short distance over which visible light is absorbed. Finally, using no tilt, the laser beams will not be projected on the seafloor.
These constraints are overcome with the method that we present in this work. The two laser lines are simply projected over the seafloor and fixed to the ROV frame, thus not rotating with the camera. In most cases the two laser lines will be within sight, although dislocated due to the camera's rotation, as described above. The use of laser lines instead of spots has the advantage of being easier to detect on still images, even in shallow water operations when scattered day light generally implies image filtering and processing to detect laser spots.
The seafloor area corresponding to a trapezoid defined by the laser lines in a still image is estimated directly from the picture by using Eq (19). Eq (22) yields the camera's height above the ground, an important variable when estimating the accuracy of conspicuous target discrimination. The lengths in pixels can be measured using any standard image analysis software.
It is important to underline that our results depend on the validity of the assumptions used for their derivation, presented in the Results and Discussion section. The mathematical solution that we obtained breaks down if either the laser lines are not parallel, the camera produces non-negligible optical distortion or the still image is cropped. Two other parameters are needed, the camera's horizontal and vertical angles of view, α H and α V , respectively (or, in fact, the corresponding nonlinear zoom-dependent curves). They are generally available in the camera specifications or can be obtained from the manufacturer.
For the in situ experiment that was performed, the test area was estimated with an error of *1.5%. The excellent agreement between the calculated and the true values strongly supports the validity of our results. On the other hand, the height of the camera relative to the seafloor was estimated with an error of *11%. Although this is still an acceptable error for most situations, the magnitude of the discrepancy is almost an order of magnitude greater than for the case of the area estimation. This is due to the fact that the value calculated for h is especially sensible to the correct alignment of the laser lines, as it explicitly depends on the "perspective" angles θ 1 and θ 2 -see Eq (22). In practice, perfect parallelism between the laser lines will always be difficult to achieve. Therefore our solution will tend to produce a larger error for h than for S. This is not problematic, as the variable of interest is the area S. If necessary, the height h may be more accurately determined with help of one of several commercially available sensors, e.g. a standard altimeter.
As explained above, it was only possible to consider one independent image. Although the results of the experiment strongly support the robustness of our solution, it is of great importance to continue to perform similar tests. With a large enough data set it will be possible to determine the precision of the method with sound statistical tools. It is of particular interest to understand the sensitivity of the method to residual misalignments of the laser lines. In practice, these will be unavoidable. The results of the experiment suggest that the estimated area, Eq (19), is not significantly affected by such residual misalignments. A large data set encompassing different control objects will also allow to understand how the accuracy of the method depends on operational parameters such as distance to target, tilt and pan angles or ROV distance from the seabed.
The main equations only involve ratios between pixel lengths measured from the still image. Thus, it might be expected that there would be no significant change to the results if the picture were to be degraded to a lower resolution, as the rescaling factors would cancel out exactly in Eqs (18)- (23). However, coarse-graining effects might not be negligible, meaning that the values of the length ratios shall be affected. In other words, the rescaling factors will not cancel exactly. Nevertheless, the uncertainty in the calculated values may be estimated by using Eqs (25)-(29). These equations show that the uncertainties increase with decreasing image resolution, i.e. when N and M decrease. In fact, as N and M decrease, the length ratios in Eqs (25)-(29) approach unity, whereas the unbalanced terms will increase due to coarse-graining (e.g. 1 / (n 0 − n@) in (25)).
The two parallel laser lines may be used as a natural scale in defining an equal seafloor area grid on the picture. By adapting the so called Canadian grid method [1], substituting the vanishing point V' for the image's optical center, it is straightforward to draw lines on the still image that correspond to virtual lines on the seafloor that are parallel to the visible laser lines.
The results presented in this work were derived under the assumption of a horizontal seabed. When such is not the case, the slope of the seafloor will be equivalent to an apparent tilt angle from the point of view of the camera. Therefore, if the actual tilt of the camera is known, the value of the apparent tilt angle will allow to correct for the inclination of the ground and to estimate the corresponding slope. The latter is a variable of great interest in benthic ecological studies and in geologic interpretation. For example, consider the camera facing a surface of inclination α, with the laser lines projected along the slope. If the camera is tilted downward by an angle λ 0 , Eq (20) gives an apparent tilt angle λ = λ 0 + α. Thereby, Eq (30) shows that the seabed slope may be estimated by measuring the true tilt angle λ 0 . This can be achieved using a pan and tilt tracking sensor coupled to the camera. If relevant, a motion reference unit (mru) moving with the ROV may also be used to account for the attitude of the vehicle.
Note that Eq (19) still applies for the area S of a surface on a sloping seabed. In fact, the derivation of Eqs (15)-(29) remains unchanged when λ is interpreted as the angle between the camera's optical axis and the seabed surface. For a sloping seafloor λ = λ 0 + α is the apparent tilt of the camera, whereas for a horizontal surface λ is the true vertical tilt angle.
The method presented in this work is independent of common sources of bias in transect methodologies for sampling the deep-sea, such as average field of view or transect distances. Therefore, it might have been useful in previous deep-sea projects that involved estimating seafloor densities, e.g. of benthic organisms [20][21][22], marine litter [23,24] or geologic structures [25]. It will be of use in future or ongoing projects (e.g. MIDAS [26] or JPI Oceans [27]), providing more accurate measurements from image data of seafloor surveys. Integration of this method into automated laser recognition routines could greatly enhance the technological capacity of seafloor assessments.