Key information

Title: Method and system for detecting a line above ground from a helicopter
Application number: 20210695
Case type: National
Status: 24.04.2023 Patent meddelt (B1)
Filed: 02.06.2021
Effective date: 02.06.2021
Publicly available: 05.12.2022
Next annual fee due: 30.06.2027
Applicant: Kleon Solutions AS (NO)
Owner: Kleon Solutions AS (NO)
Inventor: Arne Sommerfelt (NO)
Agent: ACAPO ONSAGERS AS (NO)
Granted: 24.04.2023
Patent number: 347027
Expiry date: 02.06.2041

Abstract and drawing

Disclaimer: This text has been machine-scanned and may contain errors – please refer to "Publications" for legally binding content.

The present invention relates to a method for detecting a line above ground (LAG) from a helicopter (100). The method comprises providing a first image (A) from a first viewpoint (VPA) and a second image (B) from a second viewpoint (VPB) by means of a camera (110) mounted to the helicopter (100). The first image (A) and the second image (B) have an overlapping region (OR). First, an image registration process is performed, resulting in a mapping of image points (qA) in the first image (A) to their corresponding image points (qB) in the second image (B). A ground plane (GP) and the viewpoints (VPA, VPB) are determined in a common reference system. A difference image (D) comprising all image points of the overlapping region (OR) and the difference in pixel values between the corresponding image points (qA, qB) at these image points of the overlapping region (OR) is determined. A 3D line (L3D) is determined in the common reference system based on compliant line pairs found in the difference image. A line above ground (LAG) is determined to be present if the reconstructed line (L3D) is compliant with geometric constraints formed by the viewpoints (VPA, VPB) and the ground plane (GP).

Publications

Checking availability of the latest published version...

Documents

Expand

Download files (35)

Date

Date	Doc. No.	Process	Case number	In/Out	Journal description	To/from
27.04.2023	15-01	Saksbehandling	20210695(347027)	UT	PT Registreringsbrev nasjonal patent (15) (PT20210695)
15.02.2023	14-01	Saksbehandling	20210695	UT	Intention to grant	ONSAGERS AS
15.11.2022	13-07	Saksbehandling	20210695	INN	P27597NO00 response	ONSAGERS AS
15.11.2022	13-06	Saksbehandling	20210695	INN	P27597NO00 Norwegian claims	ONSAGERS AS
15.11.2022	13-05	Saksbehandling	20210695	INN	P27597NO00 amended description markup	ONSAGERS AS
15.11.2022	13-04	Saksbehandling	20210695	INN	P27597NO00 amended description CC	ONSAGERS AS
15.11.2022	13-03	Saksbehandling	20210695	INN	P27597NO00 amended claims markup	ONSAGERS AS
15.11.2022	13-02	Saksbehandling	20210695	INN	P27597NO00 amended claims CC	ONSAGERS AS
15.11.2022	13-01	Saksbehandling	20210695	INN	Korrespondanse (Hovedbrev inn)	ONSAGERS AS
16.08.2022	12-01	Saksbehandling	20210695	UT	Substantive examination	ONSAGERS AS
11.05.2022	11-07	Saksbehandling	20210695	INN	P27597NO00 response	ONSAGERS AS
11.05.2022	11-06	Saksbehandling	20210695	INN	P27597NO00 - amended fig. 6 1053178	ONSAGERS AS
11.05.2022	11-05	Saksbehandling	20210695	INN	P27597NO00 amended description WC	ONSAGERS AS
11.05.2022	11-04	Saksbehandling	20210695	INN	P27597NO00 amended description CC	ONSAGERS AS
11.05.2022	11-03	Saksbehandling	20210695	INN	P27597NO00 amended claims WC	ONSAGERS AS
11.05.2022	11-02	Saksbehandling	20210695	INN	P27597NO00 amended claims CC	ONSAGERS AS
11.05.2022	11-01	Saksbehandling	20210695	INN	Korrespondanse (Hovedbrev inn)	ONSAGERS AS
31.03.2022	10-01	Saksbehandling	20210695	INN	Korrespondanse (Hovedbrev inn)	ONSAGERS AS
27.12.2021	09-02	Saksbehandling	20210695	UT	PT report 09:09:50
27.12.2021	09-01	Saksbehandling	20210695	UT	Substantive examination	ONSAGERS AS
27.12.2021	08-01	Saksbehandling	20210695	UT	Generelt Brev Patent	ONSAGERS AS
30.06.2021	02-01	Endring av søker/innehaver	2021/08924	UT	GH Forespørsel	ONSAGERS AS
25.06.2021	07-03	Saksbehandling	20210695	INN	P27597NO00 signert fullmakt	ONSAGERS AS
25.06.2021	07-02	Saksbehandling	20210695	INN	P27597NO00 signert erklæring	ONSAGERS AS
25.06.2021	07-01	Saksbehandling	20210695	INN	Korrespondanse (Hovedbrev inn)	ONSAGERS AS
22.06.2021	06-02	Saksbehandling	20210695	INN	P27597NO00 signert fullmakt og ekrlæring om retten til oppfinnelsen	ONSAGERS AS
22.06.2021	06-01	Saksbehandling	20210695	INN	Korrespondanse (Hovedbrev inn)	ONSAGERS AS
16.06.2021	05-01	Saksbehandling	20210695	INN	Korrespondanse (Hovedbrev inn)	ONSAGERS AS
16.06.2021	01-01	Endring av søker/innehaver	2021/08924	INN	Generell henvendelse	ONSAGERS AS
14.06.2021	04-01	Saksbehandling	20210695	INN	Korrespondanse (Hovedbrev inn)	ONSAGERS AS
07.06.2021	03-01	Saksbehandling	20210695	UT	Formal Examination 1	ONSAGERS AS
07.06.2021	02-01	Saksbehandling	20210695	UT	Infobrev til oppfinner	ONSAGERS AS
02.06.2021	01-03	Saksbehandling	20210695	INN	P27597NO00 - Patent specification final PDF 1000082	ONSAGERS AS
02.06.2021	01-02	Saksbehandling	20210695	INN	P27597NO00 - Patent drawings final PDF 1000057	ONSAGERS AS
02.06.2021	01-01	Saksbehandling	20210695	INN	Søknadsskjema Patent	ONSAGERS AS

Description, claims and drawings

Disclaimer: This text has been machine-scanned and may contain errors – please refer to "Publications" for legally binding content.

Description

FIELD OF THE INVENTION

The present invention relates to a method and system for detecting a line above ground from a helicopter.

BACKGROUND OF THE INVENTION

From time to time, fatal accidents occur due to a helicopter coming into contact with a line above ground such as power lines and telecommunication lines.

Specifically, it is a challenge to avoid such lines above ground during landing of the helicopter in terrain unfamiliar to the pilot.

Often, lines above ground have such small dimensions (small width) that they are difficult to observe with the naked eye.

Traditionally, helicopter crews have avoided lines above ground by using maps in which known lines above ground are drawn in.

Consequently, within the field, there exists a general need for methods and devices for detecting lines above ground from a helicopter automatically.

Previously, solutions have been developed to detect lines above ground from a helicopter, wherein the solution is based on use of a camera and image processing techniques processing the image shot by the camera.

For example, EP 1908 010 shows a system and a method for passive detection of lines from a helicopter. A digital image is captured from the helicopter. Firstly, the digital image is pre-processed to reduce noise which does not resemble lines. Then pixels are identified which may be classified as lines using a segment-finder module. Further, the identified pixels are linked in a linker module, to determine if a line-like structure is present. This generates a line overlay. The linker module comprises spatial and temporal filters. The spatial filter eliminates features in the image which are not consistent with lines, while the temporal filter eliminates pixels which do not fulfill a minimal image-to-image persistence. When a line is detected the pilot is warned.

In such warning systems it is desirable to improve the safety to ensure that warning is given when a line is actually present. At the same time, it is desirable to avoid a warning when an overhead cable is not present. In particular, this is the case when the line is seen with the ground below it as background. The ground may then have line-like patterns that may trigger an alarm even if they do not represent obstacles. This is a normal situation during a landing procedure in a non-regulated landing location.

NO 340705 describes a method for detecting a line above ground from a helicopter such as a helicopter having a pilot. Typically, the method is performed during a landing procedure for the helicopter. The method first comprises a step of providing, from a single camera arranged in said helicopter, a plurality of images of an area beneath said helicopter during motion of said helicopter. In a further step, the method comprises the step of detecting lines in the images, and further determining whether said lines represent lines at ground level or lines at a level above ground level. Then, the method comprises the step of determining whether a line at a level above ground level represents a possible line above ground.

The step of determining whether the lines represent lines at ground level or lines at a level above ground level comprises identifying pairs of lines belonging together from said first and second images, respectively, identifying a background in each of said first and second images, associated with the pairs of lines, and determining if a line and its associated background have shifted more than a predetermined limit from said first to said second image, and if so, determining if the line represents a line at a level above a ground level.

It is an object of the invention to provide an alternative method for detecting lines above ground from a helicopter.

SUMMARY OF THE INVENTION

The present invention relates to a method for detecting a line above ground from a helicopter during landing of the helicopter (100) in a terrain unfamiliar to the pilot, wherein the method is comprising the steps of:

- providing a first image from a first viewpoint and a second image from a second viewpoint by means of a monocular camera, wherein the camera is mounted to the helicopter and directed towards the ground below the helicopter; and wherein the helicopter is in non-linear motion between the first viewpoint and the second viewpoint and wherein the first image and the second image has an overlapping region;

- performing an image registration process resulting in a mapping of image points in the first image to their corresponding image points in the second image;

- determining a common reference system;

- determining a ground plane in the common reference system, wherein the ground plane is a planar approximation of the ground;

- determining the viewpoints in the common reference system;

- using the results of the image registration to calculate a difference image comprising all image points of the overlapping region and the difference in pixel values between the corresponding image points at these image points of the overlapping region;

- identifying any positive lines in the difference image having positive difference values;

- identifying any negative lines in the difference image having negative difference values;

- identifying the originating image for each positive line and each negative line;

- identifying any compliant line pair as two lines where one originate from the positive lines and the other originate from the negative lines and where one originates from the first image and the other originate from the second image;

- reconstructing a 3D line in the common reference system from each compliant line pair based on triangulation from the position and orientation of viewpoints;

- determining that a line above ground is present if the reconstructed line is compliant with geometric constraints formed by the viewpoints and the ground plane.

In one aspect, the method uses one camera. In one aspect, the camera is a monocular camera.

In one aspect, the step of determining that a line above ground is present further comprises the steps of:

- for each compliant line pair, for each epipolar line intersecting both lines in the compliant line pair, identifying corresponding image points in the first and second images;

- determining that a line above ground is present if the corresponding image points for the compliant line pair have compliant pixel values.

In one aspect, the steps of identifying any first lines and identifying any second lines comprises the step of:

- using a Hough transformation algorithm.

In one aspect, the method further comprises the following step before the step of performing an image registration process:

- performing a pre-processing of the first image and/or the second image to reduce the effect of variation in noise, motion blur, lighting or exposure between the images.

In one aspect, the method further comprises the following step before the step of performing an image registration process:

- performing a pre-processing of the first image and/or the second image to reduce the effect of variation in noise, motion blur, lighting or exposure between the images.

In one aspect, the step of determining the common reference system is comprising: - determining the common reference system based on the image registration process and an image projection model of the camera.

Alternatively, the common reference system may be determined by means of data from sensors resulting in a specific position and orientation of the helicopter. The sensor for measuring the specific position of the helicopter may be a GPS sensor. The sensor for measuring the orientation of the helicopter may be a gyroscope, or a magnetic compass and accelerometer, etc.

In one aspect, the method comprises a step of determining a scale of the common reference system by:

- using an altimeter; or

- using height data from a GPS sensor and map data; or

- using a distance between the first and second viewpoints by means of data from a GPS sensor.

By using the scale of the common reference system, the geometric constraints can be absolute values in meters etc. Alternatively, the geometric constraints can be relative values, for example values relative to the helicopter’s height above ground.

In one aspect, the method comprises the steps of:

- storing images from the camera in a data structure;

- selecting the first image and the second image from the data structure as two images having an overlapping region.

In one aspect, the method comprises the steps of:

- performing optical flow computations in addition to, or as part of, the image registration process.

The present invention also relates to a system for detecting a line above ground from a helicopter, wherein the system comprises:

- a monocular camera mounted to the helicopter and directed towards the ground below the helicopter;

- a signal processor connected to the camera, wherein the signal processor is configured to perform the steps of the method above.

In one aspect, the system is configured to be in communication with, or comprises: - an altimeter; or

- a GPS sensor and map data; or

- a GPS sensor;

wherein the signal processor is configured to perform the steps of the method above based on images provided by the monocular camera.

In one aspect, the system comprises:

- a pilot user interface connected to the signal processor and located in a cockpit of the helicopter, wherein the signal processor is configured to issue a warning to the pilot via the pilot user interface if it is determined that a line above ground is present.

LIST OF DRAWINGS

Fig. 1 illustrates a helicopter in two different positions during a landing operation, where a line above ground is present near the landing area;

Fig. 2a illustrates an image A, taken from the helicopter at the first position;

Fig. 2b illustrates an image B, taken from the helicopter at the second position; Fig. 2c illustrates a result of an image registration process as an image C;

Fig. 2d illustrates a difference image D;

Fig. 2e illustrates an image E resulting from a filtering transformation of the difference image D in fig. 2d that enhances the image feature of interest;

Fig. 3 illustrates the common reference system used in the method;

Fig. 4a illustrates a 3D model of the volume above ground being analyzed for a line above ground;

Fig. 4b illustrates a line above ground within a 3D model;

Fig. 4c illustrates a line in the 3D model which is not a line above ground;

Fig. 4d illustrates a line in the 3D model which is not a line above ground;

Fig. 5 illustrates a flow scheme of the steps of the method of a first embodiment. Fig. 6 illustrates the parts of a system for detecting a line above ground LAG from a helicopter 100.

DETAILED DESCRIPTION

Embodiments of the present invention will now be described in detail. Initially, some terms will be discussed.

Image representation and capture

In this description we refer to images as a central concept in our invention.

Normally the intuitive and general understanding of an image is applicable. There are however steps in our method that assume more specific details of how the image is represented. In our context an image is always a digitized two-dimensional (2D) perspective projection of a scene in the three-dimensional (3D) physical world. At each point in the image, defined by two coordinates, there is associated a pixel. The pixel describes physical properties of the corresponding point in the physical scene. Typically this will be the intensities of light reflected in specific spectral bands that together represent the color at that point in the scene. The process of creating a digital representation of an image is commonly referred to as image capture. The typical captured image is represented as a numerical tensor with 3 axes, often denoted as rows, columns and depth. There is a one-to-one mapping between rowand column indices and the image projection. The depth axis enumerates the physical property values of the pixel,- e.g. the color given as red, green and blue intensity components. Normally we do not use the term tensor, but rather describe the image as a matrix of pixels, where each pixel may be represented by multiple values. The number of pixel values is also known as the pixel depth. Examples and figures in our description will for the sake of simplicity assume single valued pixels, also known as monochrome images. There is no inherent limitation to the pixel depth that our method can use.

During image capture the typical first step is to have camera optics that project light from the scene onto a solid state sensor. The elements of the sensor matrix are also often referred to as pixels or specifically sensor pixels. Note that sensor pixels are physical elements and different from the numerically represented pixel that we refer to in our method description. Sensor pixels are normally arranged in a rectangular grid. During digitization the analogue physical properties captured by each sensor pixel are converted to numeric values, forming a raw image matrix of numerical pixels.

Camera calibration and image interpolation

The optical projection of the scene onto the sensor will in general be an approximate or distorted perspective projection because of inherent properties of the optical lens. One essential effect of this is that straight lines will not appear straight in the raw image. There are well known methods to correct the projection through a process, called camera calibration, where a mathematical model of the distortion is created. Camera calibration is done one time for each specific configuration of camera and optics. After calibration the raw image can be undistorted and thus represent a correct perspective projection of the scene. This is a standard image transformation step. Throughout the description of our method one may assume that the images are undistorted. An accurate undistorted version of the image matrix is typically created by interpolating the pixel values of the raw image matrix at non integer row/column positions determined by the distortion model. In general image interpolation allows us to look at an image as a continuously defined function at any coordinate and not only at the integer valued row/column-position of each pixel. When we refer to image points in our description, we think of these as defined by two real valued coordinates. The pixel value at an image point is computed as an interpolation of the pixel values in the image matrix that are close to the point.

Image registration

A central concept in our method is to do image registration. Image registration algorithms are considered known for a person skilled in the art. The term refers to computing a coordinate mapping function between an image point qA in image A and a point qB in image B such that qA and qB are projections of the same physical scene point. It is assumed that the images overlap, i.e. they project parts of the same scene and that the mapping is continuous and valid within the overlap. Note that the mapping function also defines the overlap since it can determine that a point qA maps to a point qB that is outside the pixel matrix of B and thus not part of the overlap.

The parameters of the mapping function is typically computed by locating and matching a sparse (eg. 10-1000 point pairs) set of corresponding points and then use this set to compute the parameters of a continuous mapping function. In its simplest form a mapping can be an affine transformation that may work well for a planar scene seen from a distance. At closer distances a homography may work better because it models the perspective transformation of a plane better than the affine transform. A more complex model can take into account local projection differences caused by a nonplanar scene. One way to do this is to compute the Essential Matrix («https://en.wikipedia.org/wiki/Essential_matrix»), E, based on use of a calibrated camera and the corresponding points found. A method to determine E from five or more corresponding points pairs is well known. The Essential Matrix describes the 3D motion of the camera and its perspective between image A and B. Given any point qA, E can be used to compute a line in B, called the epipolar line of qA, along which the corresponding point qB must be located. This allows for efficient search and computation of a registration map that models a general non-planar ground scene at the cost of more computations. This type of modelling is well established in the field of stereo vision and is often referred to as a disparity map. We want to emphasize that even if computations known from stereo vision can be exploited by our invention, it does not depend on a stereo camera. Stereo cameras capture 3D scene information instantly while our method extracts 3D scene information over time from a moving monocular camera.

One may also choose to compute a Fundamental Matrix (https://en.wikipedia.org/wiki/Fundamental_matrix_(computer_vision)) and use it in a similar manner as the Essential Matrix. This requires more corresponding points to be used, but does not require all of the intrinsic parameters of the camera model to be known. The intrinsic non-linear distortion parameters of the lens would still be needed.

There exists methods, often referred to as bundle adjustment (https://en.wikipedia.org/wiki/Bundle_adjustment), that are able to determine all intrinsic model parameters (up to scale) of a camera simply by moving the camera along a path that produces a series of overlapping images. Our method could in principle use such a method, but it is computationally expensive and would have to be completed before any obstacle detection. In this way it is similar to any other procedure used to calibrate the camera.

Similarity assumptions of registration

In practice, alle the registration mapping models described above will be based on some assumptions about the scene in relation to the resolution limit given by the image sensor-matrix. The initial set of corresponding pairs is typically found by matching a local neighbourhood of pixels in A and B. In order to do this match efficiently and reliably the corresponding neighbourhoods in A and B should have a similar scaling and it should not have different occlusions by protruding objects when seen from the different perspective of A and B. In practice this means firstly, that the change in the camera’s point of view between A and B must be limited, and secondly, that the scene must be smooth in the sense that there are only minor occlusion changes between the two points of perspective. A suitable landing ground, as observed and evaluated by the pilot in the helicopter during the landing operation, will normally comply with these criteria. This will be discussed further in detail below.

Computing the image overlap difference

The purpose of image registration in our method is to be able to compare the pixel values of two images A and B at all points in the overlap. The comparison result is computed as a difference image D, comprising image points qD of the overlapping region OR with pixel values calculated by subtracting the pixel value at point qA in the first image A from the pixel value at the corresponding image point qB in the second image B (or vice versa). Typically image D will be computed as an image matrix by use of image interpolation at corresponding points in a suitable rectangular grid.

We have explained that image registration assumes sufficient smoothness of the scene in order to succeed. When the scene is actually smooth, the image of the overlapping region in image A and image B will be identical and the pixels of image D will be close to zero or close to the inherent noise level of the system. When the scene contains non-smooth objects, like trees or powerlines, these will create significant non-zero pixel values in D that appear as image artifacts emanating from the parallax and occlusions created by different viewpoints. These image artifacts are most easily detected by computing the difference between the registered images where they overlap.

DESCRIPTION OF THE METHOD STEPS

Initially, it is referred to fig. 1, where a helicopter 100 is shown flying above ground G during a landing operation. The helicopter 100 comprises a camera 110 mounted to the helicopter 100 and directed towards the ground G below the helicopter 100. The camera 110 is a part of a system 10 (shown in fig. 5) for detecting a line above ground LAG from the helicopter 100. The camera 110 is connected to a processing unit 120 for processing images received by the camera 110. The processing unit 120 may comprise, or may be connected to, a pilot user interface 122 in the cockpit of the helicopter 100.

The present embodiment is a method for analysing images captured by the camera 110 during the landing operation and to warn the pilot via the pilot user interface 122 if the method detects that a line above ground LAG is present. Typically, most images will not contain any line above ground LAG. The example below is an example where a line above ground is present.

A line above ground LAG is here referring to a cable (such as an electric cable, a signal cable etc.), a line (a power transmission line etc.) or a rope etc. suspended from or between vertical poles, towers, etc at a distance HLAG above ground. Such lines above ground will typically be extending substantially parallel to the ground. The line above ground LAG may also be a guy rope for supporting a vertical pole, such as guy ropes for radio towers etc. Such guy ropes will typically be oriented with an angle between the radio tower and ground.

The above types of lines above ground LAG represent a risk for the helicopter 110 during the landing operation. The above types of line above ground LAG are also difficult to observe visually by the pilot, as they have a relatively thin diameter. Hence, when such lines are detected, it should result in a warning being sent to the pilot.

It should be mentioned that there may be many lines on the ground itself in images captured by the camera 110, which do not represent a risk for the helicopter 110 during the landing operation. Such lines may be road markings, curbs in urban areas, and it may be railroad tracks, thinner tree trunks in a logging area etc.

Detection of such lines should not result in a warning being sent to the pilot.

It should be mentioned that some assumptions about the landing ground conditions can be made. First, the pilot must be able to visually observe the ground G. This is a safety requirement for most landing operations with helicopters. Hence, a camera, which also relies on visibility will be applicable in all conditions where a helicopter is allowed to land. Second, it can be assumed that the ground G being analysed in the present method is generally flat and obstacle free in an area spanning typically more than two times the diameter of the helicopter rotor. Also, there will be a limit on the general inclination of the ground relative to the horizontal plane. It may further be assumed that there are no other obstacles for the helicopter to land and further that the vegetation on the ground is sufficiently small to allow the helicopter to land. In the present invention there is no assumption that the entire field of view from the camera has landing ground conditions. Plainly visible landing obstacles like trees, power line masts and buildings may be present in the field of view and may or may not be marked as obstacles by the algorithm. In these cases an obstacle warning is not essential and maybe not even desirable. As discussed above, the objective of the method is to detect these hard-to-see lines above ground LAG in the part of the field of view that appears to be a possible landing ground.

There is one limitation of the present system. If the helicopter is moving in a straight line of flight and there is a line above ground LAG parallel to the line of flight, then the method according to the present invention can not detect the line above ground LAG. In practice, this can be avoided by flying the helicopter in a curve or some other non-linear path towards a promising landing location.

In fig. 1, a first position of the helicopter is indicated by solid lines, where a first viewpoint VPA is indicating both a position for the camera 110 when a first image A is captured and a direction in which the camera 110 is oriented when the first image A is captured. This orientation may for example be expressed as a 3D vector, which will include information about yaw, roll, and pitch of the helicopter.

Consequently, the first viewpoint VPA has six degrees of freedom, three representing the position in space and three representing the orientation in space. In fig. 1, also the field of view FOVA is indicated as the area of the ground being covered by this first image A.

In fig. 1, a second position of the helicopter is indicated by dashed lines, where a second viewpoint VPB has six degrees of freedom, three representing the position in space and three representing the orientation in space. The image taken from the second viewpoint VPB is referred to as a second image B. In fig. 1, also the field of view FOVB is indicated as the area of the ground being covered by the second image B. The area of the ground being covered in both the first image A and the second image B is referred to as an overlapping region OR.

It should be noted that each of the viewpoints VPA and VPB have a limited field of view, as indicated by the triangles FOVA, FOVB in fig. 1 and indicated as pyramids in fig. 4a.

The steps of the method for detecting a line above ground LAG from the helicopter 100 will now be described in detail. In fig. 2a, an example of the first image A is shown. One known line above ground LAG is visible as a thin line in the image. The line above ground LAG is here a power line. In fig. 2b, an example of the second image B is shown. Also here the line above ground LAG is visible as a thin line in the image. It can be seen that there are similarities in first image A and the second image B. In fig. 2c, the overlapping region OR in the first image A and the second image B is shown as a dashed box. The area outside of the overlapping region OR is denoted a not overlapping region NOR.

It should also be noted that there are many lines in the first image A and the second image B. These lines are not lines above ground, they are track marks on the ground.

Computing the image overlap difference

The purpose of image registration in our context is to be able to compare two images A and B at all points in the overlap. The image registration has also been discussed in general above.

The image registration map is used to calculate a difference image D comprising image points qD of the overlapping region OR. The difference image D is calculated by subtracting the pixel value at point qA in the first image A from the pixel value at the corresponding image point qB in the second image B (or vice versa). Typically D will be computed as an image matrix by use of image interpolation at corresponding points in a suitable rectangular grid.

Fig. 2d shows the difference image D. For most of the image points in D, the pixel values will be close to zero because they represent the difference between the pixel values of identical scene points even if they are captured from two different viewpoints and at two different points in time. Zero valued pixels are shown as black in fig. 2d.

As shown in fig. 2d, there are two parallel white lines. The pixel at the image points of the upper line have positive values and is indicated as Lpos, while the pixels at image points of the bottom line have negative values and is indicated as Lneg. In addition, there are also some white dots scattered around in the image D, these may have negative or positive values not apparent in the figure, but these points will in any case be discarded by the line detection algorithm.

These lines represent the parallax created by a line above ground against its background when seen from two different viewpoints.

It should be noted that at this step of the description the method has not yet identified these lines as a line above ground LAG yet.

To improve the results of the image registration process, the method may comprise the step of performing a pre-processing of the first image A and/or the second image B to reduce the effect of variation in noise, motion blur, lighting or exposure between the images A, B.

The reference system

It is now referred to fig. 3. Here it is shown a three-dimensional reference system with two horizontal axes x and y and a vertical axis z. The vertical axis z is defined as being perpendicular to a ground plane GP which is a planar approximation of the ground G. As discussed above, it can be assumed that the ground is relatively flat.

In fig. 3 it is also shown that the first viewpoint VPA is selected as origo in the reference system.

The practical importance of choosing this reference system is that it makes it simple to quantify the height above ground for a line above ground when its coordinates are given in this reference system.

It is now referred to Fig. 4a. Here, the ground G is shown to be even relative to the height of the helicopter as indicated by the first viewpoint VPA and the second viewpoint VPB. However, the ground G is not shown to be entirely flat. This relatively even ground G is shown in Fig. 4a for illustrative purposes only, to illustrate the assumption made above that the ground G is even as it is a possible landing area identified by the pilot. In the context of the geometric model of the complete scene including the camera movement we assume the ground to be represented as a geometric plane that is an approximation of the landing ground.

Each viewpoint VPA, VPB is here illustrated as a pyramid (with dashed lines), while the overlapping space between the two viewpoints VPA, VPB is indicated by solid lines. The base of the overlapping space corresponds to the overlapping region OR.

The results of the above image registration process are now used to determine the viewpoints (i.e. location and orientation of the viewpoints) VPA, VPB in the common reference system of fig. 3. This is performed by decomposing the above essential matrix and triangulating ground points found during the image registration process. In addition, an image projection model of the camera 110 may be used. The image projection model is a model containing properties of the lens of the camera and the distance between the lens and the sensor of the camera.

It should be noted that a fundamental limitation to 3D reconstruction using a monocular camera, as here, is that the scale of the model is unknown. The scale is not needed to find or determine that a line above ground is present. It is only needed to quantify how far above ground the line is and compare it to a height threshold in order to decide whether it represents an obstacle or not. In practice uncertainty can be accepted in the scale input at the cost of a slightly higher risk of alerting on obstacles that are below the set detection height threshold. If for example the desired detection threshold is set to 1 m above ground and the scale input is ±25% accurate the detection threshold can be adjusted to 0.75 m to allow for the uncertainty of the scale. Possible ways to set scale from external input include any single or combined input of the following:

● A measure of the height above ground, possibly computed from map data, a ground radar, a stereo-camera, an altimeter or even the pilot’s assessment. As shown in fig. 5, the system 10 comprises a GPS sensor 130 and map information 132 from a map navigation system to set scale. Alternatively, as shown in fig. 5, the system 10 may comprise an altimeter 131 to set scale. ● A measure of the distance travelled by the helicopter between VPA and VPB, which can be extracted from an on-board inertial- or GPS-instrument. ● A measure of distance between any two points on the ground seen by the camera in a single image.

It should be noted that the present invention will be able to determine that a line above ground LAG is present without scaling of the common reference system. The above threshold can be set as a relative value, for example as 2% of the helicopter’s height above ground, and the helicopter is 100 m above ground, it means that lines 2m or more above ground will result in a warning. If the helicopter subsequently descends to 50 m without changing the relative threshold, any line above 1 m will give a warning. Thus, we see that a relative threshold can have a desirable effect in becoming more conservative as the helicopter approaches the ground.

It is now referred to fig. 4b again. If the first viewpoint VPA was selected as origo in the common reference system, the ground plane GP may be determined as a planar approximation of the ground G. This approximation can be computed by triangulating corresponding points (qA,qB) in image A and B, and thus determine their 3D position relative to VPA. Several numerical optimization methods are available for fitting the plane parameters to the set of points by minimizing a suitable cost function. Robust techniques like RANSAC (https://en.wikipedia.org/wiki/Random_sample_consensus), are able to ignore points that deviate significantly from the plane, e.g. points on nearby trees bordering the landing ground, and is hence one example of a numerical optimization method which is suitable for determining the ground plane GP in the common reference system.

Line identification

In a next step, the difference image D is analysed to identify lines in the difference image D. It should be noted that a line can extend over the entire difference image D. It should be noted that the term line may also refer to a line segment extending only over a part of the difference image D.

Algorithms for identifying line segments in an image are known for a person skilled in the art. One such algorithm is the Hough transformation algorithm.

Because of noise in the sensor data and registration inaccuracies the initial difference image will have mostly non-zero pixel values. The noise level will have local variations in its magnitude due to registration inaccuracies and ground texture properties. Thus pre-processing is performed on the difference image to normalize the noise level in a way that a uniform threshold equal to 1 can be used to create a binary image where only significant difference pixels are non-zero. Each pixel of the significant difference image will have a depth of two, where the first value is non-zero if and only if the normalized difference is above 1 and the second value is non-zero only if the normalized difference is below -1. The significant difference image D shown in fig. 2d and shows a white dot where either one of the two pixel components is non-zero. The figure thus does not distinguish negative from positive significant pixel. For the line-like patterns clearly seen in the figure this is instead indicated by the Lpos/Lneg labels. In fig. 2e we illustrate the result of the line detection based on input from D. This image is not actually produced by the detection algorithm rather it is shown here to show the noise reducing effect of the line detection. The line-like structures have been identified and are shown as two solid lines while all the other significant pixels in D have been identified as noise and consequently ignored.

Positive lines Lpos in the difference image are identified solely from the first value of each pixel in D. Then any negative lines Lneg are identified solely from the second value of each pixel in D.

It must also be determined whether a line originates from image A or B. This cannot be determined from D. Instead the corresponding pixels of A and B that correspond with the line must be examined. Assume a positive line has been found in D, then it must either originate from a dark line against a lighter background in A or a light line against a darker background in B. We use the registration to map the line-points back to the A and B images in turn and compute the average pixel value difference between the line points and their local surroundings. The image origin (A or B) of the line is then selected based on where the average line-background difference best matches the difference found in D.

In the present example shown in fig. 2e, the positive line Lpos is originating from the second image B, while the negative line Lneg is originating from the first image A. Consequently, the positive line Lpos from fig. 2d is indicated as Lpos(from B) in fig. 2e, and the negative line Lneg from fig. 2d is indicated as Lneg(from A) in fig.

2e.

Then, compliant line pairs are identified. A compliant line pair is defined as two lines where one originate from the positive lines Lpos and the other originate from the negative lines Lneg and where one originates from the first image A and the other originate from the second image B.

Hence, two positive lines are not a compliant line pair, two negative lines are not a compliant line pair. Similarly, one positive line originating from image A and a negative line originating from image A are not a compliant line pair. It should be noted that the position, direction, width or length of the lines forming a compliant line pair are not evaluated or analysed at this stage.

If or when a compliant line pair is identified, a 3D line referred to as L3D is reconstructed in the common reference system from each compliant line pair based on triangulation from the viewpoints VPA, VPB.

One such reconstructed line L3D is shown in fig. 4b. Here, the reconstructed line L3D has been reconstructed from a compliant line pair formed by the positive line Lpos (from A) and the negative line Lneg (from B).

In a final step, the reconstructed line L3D is analysed. It is determined that a line above ground LAG is present if the reconstructed line L3D is compliant with geometric constraints formed by the viewpoints VPA, VPB, the ground plane GP and the common reference system.

In fig. 4b, the line L3D is considered to be compliant with geometric constraints, as the line L3D has a height above the ground plane GP and below the first and second viewpoints VPA, VPB.

A further geometric constraint fulfilled by the line L3D is that it is substantially parallel with the ground plane VPA. It should be noted that this is not necessarily a required geometric constraint, as the method may be used to identify guy ropes for radio towers etc. as discussed above.

It is now referred to fig. 4c where a geometrically non-compliant line-pair is illustrated. A positive line Lpos, is shown as a dotted line projected onto the ground plane, and a negative line Lneg, shown as a dashed line projected onto the ground plane. The reconstructed line L3D, does not violate the constraints formed by the viewpoints and the ground plane. Note that in our figure the epipolar lines are parallel to the horizontal grid lines shown on the ground plane because the movement of the viewpoint is parallel to these lines. We then determine the set of corresponding point pairs (pA on Lpos, and pB on Lneg) that intersect the same epipolar line. Points on any of the observed lines (Lpos or Lneg) that are not part of this set of corresponding pairs cannot belong to the same physical line in the scene and thus indicates that the lines are not geometrically compliant. The segment of Lpos that has epipolar correspondence with Lneg is illustrated as a dotted triangle. Likewise, the segment of Lneg that has epipolar correspondence with Lpos is illustrated as a dashed triangle. We then notice that the upper-right part of Lpos and lower left part of Lneg do not have correspondence. Since these parts are substantial they cannot be explained by noise. This leads to the conclusion that the lines are not geometrically compliant.

It is now referred to fig. 4d, where another geometrically non-compliant line-pair is illustrated. In this case the reconstructed 3D line (L3D) is determined to be below the ground plane, which is not possible in the physical scene.

Claims

CLAIMS

1. Method for detecting a line above ground (LAG) from a helicopter (100) during landing of the helicopter (100) in a terrain unfamiliar to the pilot, wherein the method is comprising the steps of:

- providing a first image (A) from a first viewpoint (VPA) and a second image (B) from a second viewpoint (VPB) by means of a monocular camera (110), wherein the camera (110) is mounted to the helicopter (100) and directed towards the ground (G) below the helicopter (100); and wherein the helicopter (100) is in non-linear motion between the first viewpoint (VPA) and the second viewpoint (VPB) and wherein the first image (A) and the second image (B) has an overlapping region (OR);

- performing an image registration process resulting in a mapping of image points (qA) in the first image (A) to their corresponding image points (qB) in the second image (B);

- determining a common reference system;

- determining a ground plane (GP) in the common reference system, wherein the ground plane (GP) is a planar approximation of the ground (G);

- determining the viewpoints (VPA, VPB) in the common reference system;

- using the results of the image registration to calculate a difference image (D) comprising all image points of the overlapping region (OR) and the difference in pixel values between the corresponding image points (qA, qB) at these image points of the overlapping region (OR);

- identifying any positive lines (Lpos) in the difference image (D) having positive difference values;

- identifying any negative lines (Lneg) in the difference image (D) having negative difference values;

- identifying the originating image (A, B) for each positive line (Lpos) and each negative line (Lneg);

- identifying any compliant line pair as two lines where one originate from the positive lines (Lpos) and the other originate from the negative lines (Lneg) and where one originates from the first image (A) and the other originate from the second image (B);

- reconstructing a 3D line (L3D) in the common reference system from each compliant line pair based on triangulation from a position and an orientation of the viewpoints (VPA, VPB);

- determining that a line above ground (LAG) is present if the reconstructed line (L3D) is compliant with geometric constraints formed by the viewpoints (VPA, VPB) and the ground plane (GP).

2. Method according to claim 1, wherein the step of determining that a line above ground (LAG) is present further comprises the steps of:

- for each compliant line pair, for each epipolar line intersecting both lines in the compliant line pair, identifying corresponding image points (qA, qB) in the first and second images (A, B);

- determining that a line above ground (LAG) is present if the corresponding image points for the compliant line pair have compliant pixel values.

3. Method according to any one of the above claims, wherein the steps of identifying any first lines (Lpos) and identifying any second lines (Lneg) comprises the step of:

- using a Hough transformation algorithm.

4. Method according to any one of the above claims, wherein the method further comprises the following step before the step of performing an image registration process:

- performing a pre-processing of the first image (A) and/or the second image (B) to reduce the effect of variation in noise, motion blur, lighting or exposure between the images (A, B).

5. Method according to any one of the above claims, wherein the step of determining the common reference system is comprising:

- determining the common reference system based on the image registration process and an image projection model of the camera (110).

6. Method according to any one of the above claims, wherein the method comprises a step of determining a scale of the common reference system by:

- using an altimeter; or

- using height data from a GPS sensor and map data; or

- using a distance between the first and second viewpoints (VPA, VPB) by means of data from a GPS sensor.

7. Method according to any one of the above claims, wherein the method comprises the steps of:

- storing images from the camera (110) in a data structure;

- selecting the first image (A) and the second image (B) from the data structure as two images having an overlapping region (OR).

8. Method according to any one of the above claims, wherein the method comprises the steps of:

- performing optical flow computations in addition to, or as part of, the image registration process.

9. System (10) for detecting a line above ground (LAG) from a helicopter (100), wherein the system (10) comprises:

- a monocular camera (110) mounted to the helicopter (100) and directed towards the ground (G) below the helicopter (100);

- a signal processor (120) connected to the camera (110), wherein the signal processor (120) is configured to perform the steps of the method according to claims 1 –5 and/or claims 7 - 8 based on images provided by the monocular camera (110).

10. System (10) according to claim 9, wherein the system (10) is configured to be in communication with, or comprises:

- an altimeter; or

- a GPS sensor and map data; or

- a GPS sensor;

wherein the signal processor (120) is configured to perform the steps of the method according to claim 6 based on images provided by the monocular camera (110).

11. System (10) according to claim 9 or 10, wherein the system (10) comprises: - a pilot user interface (122) connected to the signal processor (120) and located in a cockpit of the helicopter (100), wherein the signal processor (120) is configured to issue a warning to the pilot via the pilot user interface (122) if it is determined that a line above ground (LAG) is present.

PATENTKRAV

1. Fremgangsmåte for å detektere en linje over bakken (LAG) fra et helikopter (100) under landing av helikopteret (100) i et terreng som er ukjent for piloten, hvori metoden omfatter trinnene:

- å tilveiebringe et første bilde (A) fra et første synspunkt (VPA) og et andre bilde (B) fra et andre synspunkt (VPB) ved hjelp av et monokulært kamera (110), hvori kameraet (110) er montert på helikopteret (100) og rettet mot bakken (G) under helikopteret (100); og hvori helikopteret (100) er i ikke-lineær bevegelse mellom det første synspunktet (VPA) og det andre synspunktet (VPB) og hvori det første bildet (A) og det andre bildet (B) har et overlappende område (OR);

- å utføre en bilderegistreringsprosess som resulterer i en kartlegging av bildepunkter (qA) i det første bildet (A) til deres tilsvarende bildepunkter (qB) i det andre bildet (B);

- å bestemme et felles referansesystem;

- å bestemme et jordplan (GP) i det felles referansesystemet, hvori jordplanet (GP) er en plan tilnærming av bakken (G);

- å bestemme synspunktene (VPA, VPB) i det felles referansesystemet;

- å bruke resultatene av bilderegistreringen for å beregne et forskjellsbilde (D) som omfatter alle bildepunktene i det overlappende området (OR) og forskjellen i pikselverdier mellom de tilsvarende bildepunktene (qA, qB) ved disse bildepunktene i den overlappende regionen (OR);

- å identifisere eventuelle positive linjer (Lpos) i forskjellsbildet (D) som har positive forskjellsverdier;

- identifisere eventuelle negative linjer (Lneg) i forskjellsbildet (D) som har negative forskjellsverdier;

- å identifisere det opprinnelige bildet (A, B) for hver positiv linje (Lpos) og hver negativ linje (Lneg);

- å identifisere ethvert kompatibelt linjepar som to linjer der den ene stammer fra de positive linjene (Lpos) og den andre stammer fra de negative linjene (Lneg) og hvor den ene kommer fra det første bildet (A) og den andre kommer fra det andre bildet (B);

- å rekonstruere en 3D-linje (L3D) i det felles referansesystemet fra hvert kompatibelt linjepar basert på triangulering fra en posisjon og en orientering av synspunktene (VPA, VPB);

- å fastslå at en linje over bakken (LAG) er tilstede dersom den rekonstruerte linjen (L3D) er i samsvar med geometriske begrensninger dannet av utsiktspunktene (VPA, VPB) og bakkeplanet (GP).

2. Fremgangsmåte ifølge krav 1, hvori trinnet med å bestemme at en linje over bakken (LAG) er tilstede, videre omfatter trinnene:

- for hvert ettergivende linjepar, for hver epipolar linje som skjærer begge linjene i det ettergivende linjeparet, å identifisere tilsvarende bildepunkter (qA, qB) i det første og andre bildet (A, B);

- å bestemme at en linje over bakken (LAG) er tilstede hvis de tilsvarende bildepunktene for det kompatible linjeparet har kompatible pikselverdier.

3. Fremgangsmåte ifølge hvilket som helst av de ovennevnte kravene, hvori trinnene med å identifisere eventuelle første linjer (Lpos) og identifisere eventuelle andre linjer (Lneg) omfatter trinnet:

- å bruke en Hough-transformasjonsalgoritme.

4. Fremgangsmåte ifølge hvilket som helst av de ovennevnte kravene, hvori fremgangsmåten videre omfatter følgende trinn før trinnet med å utføre en bilderegistreringsprosess:

- å utføre en forbehandling av det første bildet (A) og/eller det andre bildet (B) for å redusere effekten av variasjon i støy, bevegelsesuskarphet, belysning eller eksponering mellom bildene (A, B).

5. Fremgangsmåte ifølge hvilket som helst av de ovennevnte kravene, hvori trinnet med å bestemme det felles referansesystemet omfatter:

- å bestemme det felles referansesystemet basert på bilderegistreringsprosessen og en bildeprojeksjonsmodell av kameraet (110).

6. Fremgangsmåte ifølge hvilket som helst av de ovennevnte kravene, hvori fremgangsmåten omfatter et trinn med å bestemme en skala for det felles referansesystemet ved:

- bruk av høydemåler; eller

- bruk av høydedata fra en GPS-sensor og kartdata; eller

- bruk av avstand mellom første og andre synspunkt (VPA, VPB) ved hjelp av data fra en GPS-sensor.

7. Fremgangsmåte ifølge hvilket som helst av de ovennevnte kravene, hvori fremgangsmåten omfatter trinnene:

- å lagre bilder fra kameraet (110) i en datastruktur;

- å velge det første bildet (A) og det andre bildet (B) fra datastrukturen som to bilder som har et overlappende område (OR).

8. Fremgangsmåte ifølge hvilket som helst av de ovennevnte kravene, hvori fremgangsmåten omfatter trinnene:

- å utføre optiske flytberegninger i tillegg til, eller som en del av, bilderegistreringsprosessen.

9. System (10) for å detektere en linje over bakken (LAG) fra et helikopter (100), hvori systemet (10) omfatter:

- et monokulært kamera (110) montert på helikopteret (100) og rettet mot bakken (G) under helikopteret (100);

- en signalprosessor (120) koblet til kameraet (110), hvori signalprosessoren (120) er konfigurert til å utføre trinnene i fremgangsmåten ifølge kravene 1-5 og/eller kravene 7-8 basert på bilder levert av det monokulære kameraet (110).

10. System (10) ifølge krav 9, hvori systemet (10) er konfigurert til å være i kommunikasjon med, eller omfatter:

- en høydemåler; eller

- en GPS-sensor og kartdata; eller

- en GPS-sensor;

hvori signalprosessoren (120) er konfigurert til å utføre trinnene i fremgangsmåten ifølge krav 6 basert på bilder levert av det monokulære kameraet (110).

11. System (10) ifølge krav 9 eller 10, hvori systemet (10) omfatter:

- et pilotbrukergrensesnitt (122) koblet til signalprosessoren (120) og plassert i en cockpit på helikopteret (100), hvori signalprosessoren (120) er konfigurert til å gi en advarsel til piloten via pilotbrukergrensesnittet (122) dersom det fastslås at en linje over bakken (LAG) er tilstede.

Drawings

Priority

No priority

Patent classes

IPC classes

G06T 7/215

G06T 7/246

G06T 7/254

G06T 7/55

G06T 7/579

G01C 11/30

G01C 11/34

CPC classes

G06T 7/215

G06T 7/246

G06T 7/254

G06T 7/55

G06T 7/579

G01C 11/30

G01C 11/34

G06T 2207/30184

Citations

EP 3238173 B1 (B1)

Ye Lu et. al. A Survey of Motion-Parallax-Based 3-D Reconstruction Algorithms, in IEEE Transactions on systems, man and cybernetics, part C: Applications and Reviews, vol 34, no. 4 November 2004. DOI 10.1109/TSMCC.2004.829300 ()

Schenk, T. From point-based to feature based aerial triangulation. In: ISPRS Journal of Photogrammetry and Remote Sensing 58 (2004) p. 315-329, Elsevier, DOI: 10.1016/j.isprsjprs.2004.02.003. ()

Applicants, owners and inventors

Owners

Kleon Solutions AS

Korsavegen 7B

6809 FØRDE

NO (SUNNFJORD Municipality, Vestland county)

Org. number: 999599532

Inventors

Arne Sommerfelt

Dalsbergstien 8

0170 OSLO

NO (OSLO Municipality, Oslo county)

Agent

ACAPO ONSAGERS AS

Edvard Griegs vei 1

5059 BERGEN

NO (BERGEN Municipality, Vestland county)

Org. number: 932214636

Reference: P27597NO00

Status history

Payment history

Oppositions and appeals

No oppositions or appeals

Administrative limitation

No administrative limitation

Liens and licences

No liens or licenses registered

Gazette entries

No Gazette entries found

Only events from 2024.03.04 and onwards are shown.

Method and system for detecting a line above ground from a helicopter