Key information

Title
Augmented reality system and method of displaying an augmented reality image
Application number
20171008
Case type
National
Status
09.11.2020 Opphevet etter innsigelse
Filed
20.06.2017
Effective date
20.06.2017
Publicly available
06.08.2018
Applicant
Augmenti AS (NO)
Owner
Augmenti AS (NO)
Inventor
Kenneth Gade (NO) et al.
Agent
Valea AB (SE)
Granted
06.08.2018
Patent number
342793

Abstract and drawing


Disclaimer: This text has been machine-scanned and may contain errors – please refer to "Publications" for legally binding content.
The present disclosure regardsan augmented reality system (100), comprising a global navigation satellite system module (101) adapted to output position data, an orientation measurementmodule (102) adapted to output orientation data, an augmented reality module (103), at least one AR-Client (104a, 104b) comprisinga camera(1040a, 1040b) and a display (1041a, 1041b), wherein the augmented reality module (103) is adapted to determine a position and orientation of the camera (1040a, 1040b) of the at least one AR-client (104a, 104b) based on the position data and orientation data, calculating screen positions of at least one AR object (201 a, 201 b) based on the position and orientation of the camera (1040a, 1040b) of the at least one AR-client (104a, 104b) to create at least one ARoveriay (200), transmitting the at least one AR overlay (200) to at least one ARclient (104a, 104b), and the AR-client (104a, 104b) is adapted to merging the at least one AR-overlay (200) with a picture (203) received from the camera (1040a, 1040b) of the at least one AR-client (104a, 104b) to provide an AR-image (204), and displaying the AR-image (204) on the display (1041 a, 1041 b). In addition, the present disclosure relates to a method of displaying an augmented reality image.

Publications


B1
Checking availability of the latest published version...

Documents


Expand Download files (84)
Date
Date Doc. No. Process Case number In/Out Journal description To/from
09.11.2020 25-03 Innsigelse 2019/00190 UT O70082NO00_Svar_Patentstyrets_uttalelse_Nov_03,_2020
09.11.2020 25-02 Innsigelse 2019/00190 UT IAIN07P- Avgjørelse - innsigelse_1350907
09.11.2020 25-01 Innsigelse 2019/00190 UT OP09: Oversendelse med avgjørelse Valea AB
09.11.2020 24-03 Innsigelse 2019/00190 UT O70082NO00_Svar_Patentstyrets_uttalelse_Nov_03,_2020
09.11.2020 24-02 Innsigelse 2019/00190 UT IAIN07P- Avgjørelse - innsigelse_1350907
09.11.2020 24-01 Innsigelse 2019/00190 UT OP09: Oversendelse med avgjørelse HÅMSØ PATENTBYRÅ AS
03.11.2020 23-02 Innsigelse 2019/00190 INN O70082NO00_Svar_Patentstyrets_uttalelse_Nov_03,_2020
03.11.2020 23-01 Innsigelse 2019/00190 INN Korrespondanse (Hovedbrev inn)
28.08.2020 22-01 Innsigelse 2019/00190 UT Opp til avgjørelse Valea AB
28.08.2020 21-01 Innsigelse 2019/00190 UT Opp til avgjørelse HÅMSØ PATENTBYRÅ AS
06.08.2020 20-01 Innsigelse 2019/00190 UT Oversendelse Valea AB
06.08.2020 19-02 Innsigelse 2019/00190 UT Hovedbrev, vedlegg
06.08.2020 19-01 Innsigelse 2019/00190 UT Oversendelse HÅMSØ PATENTBYRÅ AS
29.06.2020 18-02 Innsigelse 2019/00190 INN Hovedbrev, vedlegg
29.06.2020 18-01 Innsigelse 2019/00190 INN Korrespondanse (Hovedbrev inn)
18.06.2020 17-01 Innsigelse 2019/00190 UT Fristforespørsel informasjonsbrev Valea AB
18.06.2020 16-01 Innsigelse 2019/00190 UT Fristforlengelse - Informasjonsbrev til motpart HÅMSØ PATENTBYRÅ AS
29.04.2020 15-01 Innsigelse 2019/00190 INN Korrespondanse (Hovedbrev inn)
29.04.2020 14-01 Innsigelse 2019/00190 INN Korrespondanse (Hovedbrev inn) DEHNS AS
31.01.2020 13-01 Innsigelse 2019/00190 UT Generelt brev HÅMSØ PATENTBYRÅ AS
30.01.2020 12-01 Innsigelse 2019/00190 UT Oversendelse HÅMSØ PATENTBYRÅ AS
30.01.2020 11-06 Innsigelse 2019/00190 UT D17-GB2477787B
30.01.2020 11-05 Innsigelse 2019/00190 UT D16-epo-WO
30.01.2020 11-04 Innsigelse 2019/00190 UT Hovedbrev
30.01.2020 11-03 Innsigelse 2019/00190 UT D18-US20160048203A1
30.01.2020 11-02 Innsigelse 2019/00190 UT Korrespondanse (Hovedbrev inn)
30.01.2020 11-01 Innsigelse 2019/00190 UT Oversendelse Valea AB
22.01.2020 10-06 Innsigelse 2019/00190 INN D18-US20160048203A1
22.01.2020 10-05 Innsigelse 2019/00190 INN D17-GB2477787B
22.01.2020 10-04 Innsigelse 2019/00190 INN D19-brookshire-iitsec2015
22.01.2020 10-03 Innsigelse 2019/00190 INN D16-epo-WO
22.01.2020 10-02 Innsigelse 2019/00190 INN Hovedbrev
22.01.2020 10-01 Innsigelse 2019/00190 INN Korrespondanse (Hovedbrev inn)
28.11.2019 09-01 Innsigelse 2019/00190 UT Fristforlengelse - Informasjonsbrev til motpart Valea AB
28.11.2019 08-01 Innsigelse 2019/00190 UT Fristforespørsel informasjonsbrev HÅMSØ PATENTBYRÅ AS
21.11.2019 07-01 Innsigelse 2019/00190 INN Korrespondanse (Hovedbrev inn)
22.08.2019 06-01 Innsigelse 2019/00190 UT Oversendelse Valea AB
22.08.2019 05-02 Innsigelse 2019/00190 UT Hovedbrev
22.08.2019 05-01 Innsigelse 2019/00190 UT Oversendelse HÅMSØ PATENTBYRÅ AS
13.08.2019 04-03 Innsigelse 2019/00190 INN Hovedbrev DEHNS AS
13.08.2019 04-02 Innsigelse 2019/00190 INN Fullmakt DEHNS AS
13.08.2019 04-01 Innsigelse 2019/00190 INN Korrespondanse (Hovedbrev inn) DEHNS AS
22.05.2019 03-01 Innsigelse 2019/00190 UT Bekreftelse HÅMSØ PATENTBYRÅ AS
22.05.2019 02-01 Innsigelse 2019/00190 UT Oversendelse Valea AB
07.05.2019 16-01 Saksbehandling 20171008 UT PT Varsel om betaling av første årsavgift (3317) (PT20171008)
06.05.2019 01-14 Innsigelse 2019/00190 INN D5 - US7046214B2
06.05.2019 01-13 Innsigelse 2019/00190 INN D6 - WO03060830A1
06.05.2019 01-12 Innsigelse 2019/00190 INN D7 - EN_delvis_sertifisert
06.05.2019 01-11 Innsigelse 2019/00190 INN D2 - MP-AVT-256-02
06.05.2019 01-10 Innsigelse 2019/00190 INN D3 - Seven Ways to Find Heading
06.05.2019 01-09 Innsigelse 2019/00190 INN D4 - US2003210228A1
06.05.2019 01-08 Innsigelse 2019/00190 INN D7 - WO2009138069A1
06.05.2019 01-07 Innsigelse 2019/00190 INN D10 - WO2017020132A1
06.05.2019 01-06 Innsigelse 2019/00190 INN D9 - US2016132046A1
06.05.2019 01-05 Innsigelse 2019/00190 INN Hovedbrev
06.05.2019 01-04 Innsigelse 2019/00190 INN D7 - Sertifisering av oversettelse
06.05.2019 01-03 Innsigelse 2019/00190 INN D7 - WO2009138069A1_EN_Espacenet
06.05.2019 01-02 Innsigelse 2019/00190 INN D8 - US2011115816A1
06.05.2019 01-01 Innsigelse 2019/00190 INN Nytt overprøvningskrav (OP)
06.08.2018 15-01 Saksbehandling 20171008 UT PT Registreringsbrev Nasjonal Patent (15)
04.06.2018 14-01 Saksbehandling 20171008 UT Intention to grant Valea AB
08.02.2018 13-01 Saksbehandling 20171008 INN Korrespondanse (Hovedbrev inn) Valea AB
08.02.2018 12-01 Saksbehandling 20171008 UT Bekreftelse på patentsøknad Valea AB
12.01.2018 11-01 Saksbehandling 20171008 UT Generelt Brev Patent Valea AB
18.12.2017 10-05 Saksbehandling 20171008 INN Krav Valea AB
18.12.2017 10-04 Saksbehandling 20171008 INN Beskrivelse ME Valea AB
18.12.2017 10-03 Saksbehandling 20171008 INN Beskrivelse Valea AB
18.12.2017 10-02 Saksbehandling 20171008 INN Hovedbrev Valea AB
18.12.2017 10-01 Saksbehandling 20171008 INN Korrespondanse (Hovedbrev inn) Valea AB
21.11.2017 09-02 Saksbehandling 20171008 UT PT report 01:35:29
21.11.2017 09-01 Saksbehandling 20171008 UT Realitet patent Valea AB
27.09.2017 08-01 Saksbehandling 20171008 UT Generelt engelsk brev Valea AB
15.08.2017 07-01 Saksbehandling 20171008 INN Korrespondanse (Hovedbrev inn) Valea AB
08.08.2017 06-01 Saksbehandling 20171008 UT Henlagt manglede svar EN Valea AB
03.08.2017 05-02 Saksbehandling 20171008 INN Fullmakt Valea AB
03.08.2017 05-01 Saksbehandling 20171008 INN Korrespondanse (Hovedbrev inn) Valea AB
03.08.2017 04-02 Saksbehandling 20171008 INN Erklæring Søkers rett til oppfinnelse Valea AB
03.08.2017 04-01 Saksbehandling 20171008 INN Korrespondanse (Hovedbrev inn) Valea AB
22.06.2017 03-01 Saksbehandling 20171008 UT Formal Examination 1 Valea AB
22.06.2017 02-01 Saksbehandling 20171008 UT Infobrev til oppfinner Valea AB
20.06.2017 01-04 Saksbehandling 20171008 INN Beskrivelse Valea AB
20.06.2017 01-03 Saksbehandling 20171008 INN Patenttegninger Valea AB
20.06.2017 01-02 Saksbehandling 20171008 INN PT søknadsskjema, tilleggsskjema flere oppfinnere Valea AB
20.06.2017 01-01 Saksbehandling 20171008 INN Søknadsskjema Patent Valea AB

Description, claims and drawings


Disclaimer: This text has been machine-scanned and may contain errors – please refer to "Publications" for legally binding content. Description
TitleAugmented reality system and method of displaying an augmented reality imageField of the inventionThe invention relates to augmented reality systems and methods of displaying an augmented reality image.Background of the inventionAugmented Reality (AR), also known as Mixed Reality, augments what a user sees by superimposing virtual information on the real world view as seen from the user. AR is used to place virtual objects that are geo-positioned in the real world into the view of a user such that the user sees the augmented virtual object where it belongs in the real world. That is, when the user for instance looks at a real person with a known position, a virtual object representing that person should be seen by the user as positioned aligned to that person, even when both the user and the person moves around.To place virtual objects with high precision require accurate measurements of the direction of the user’s view. Current AR-systems having high directional accuracy are dependent of an external navigational unit and a separate computer that runs the AR-software. The use of two units increases the size, complexity, and cost.Many technologies exist that may provide a direction, for example magnetic compass, gyro compass, and combining several Global Navigation Satellite System (GNSS) antennas. The direction given by a magnetic compass is influenced by global and local magnetic fields and it cannot be used to place AR-symbols with sufficient accuracy and reliability. There are many sources of magnetic disturbances, e.g. natural and man-made magnetic material in the surroundings and electromagnetic interference from the equipment (for instance a vehicle) that the compass is attached to. A gyro compass finds true north with good accuracy without being affected by magnetic fields. Its disadvantages are size, weight, cost, and start up time of finding initial heading. The gyro compass also has a reduced heading accuracy when far north or south (since the horizontal component of the earth rotation vector is becoming short at high latitudes).Combining the output from several GNSS antennas can be used to give the direction with good accuracy. This requires a necessary distance between the antennas which makes it unsuitable for a large number of AR-solutions.Current units that produce high accuracy direction are based on gyro compasses, and are too big, too costly, and draw too much power, to be used in anything but high cost solutions. Current low cost systems struggle to achieve sufficient accuracy in estimating the heading. For example, mobile phones are often equipped with both a GNSS solution and a digital magnetic compass. Solutions with AR-software running on these phones depending on the digital magnetic compass for direction are not able to place their AR-symbols with required accuracy due to the error in the direction.Current AR-systems have a direct connection between the computational unit and the display unit (independently if this is using see-through or video-seethrough or other means of displaying the AR-content). That is, the display unit is connected with a cable as when using a desktop computer with a monitor or the display is built-in as in a tablet.Current AR-systems will normally produce AR-images for one particular configuration for one particular kind of user, e.g. a user of a HMD unit. This image may in some cases be shared with others, but it is then the same AR-image that all the users will see. GB2477787 disclose a portable electronic device comprising a first input for receiving a first image signal from a camera of a head-mounted display (HMD) device, a second input for receiving a position signal from a position sensor in proximity to the portable electronic device, and a third input for receiving a first orientation signal from an orientation sensor of the HMD device; a processor for running an augmented reality (AR) application to generate a data overlay of the first image signal based on the position signal and the first orientation signal and transmit the data overlay 32 to a display of the headmounted display device.The invention provides high precision placement of AR-symbols by combining a high precision directional unit and AR-software that feeds one or several distributed AR-clients with AR-overlays in one compact common unit.Brief summary of the inventionAccording to a first aspect of the embodiments herein it is provided an augmented reality system, comprising a global navigation satellite system module adapted to output position data, an orientation measurement module adapted to output orientation data, an augmented reality module, at least one AR-client comprising a camera and a display, wherein the augmented reality module is adapted todetermine a position and orientation of the camera of the at least one AR-client based on the position data and orientation data, calculating screen positions of at least one AR object based on the position and orientation of the camera of the at least one AR-client to create at least one AR-overlay, transmitting the at least one AR overlay to at least one AR-client, and the AR-client is adapted to merging the at least one AR-overlay with a picture received from the camera of the at least one AR-client to provide an AR-image, and displaying the AR-image on the display. According to a second aspect of the embodiments herein it is provided a method of displaying an augmented reality image, comprising determining a position of an augmented reality device by a global navigation satellite system, measuring an orientation of the augmented reality device by an orientation measurement module, determining, based on the position and orientation of the augmented reality device, the position and orientation of at least one camera, calculating screen positions of at least one AR object based on the position and orientation of the at least one camera to create at least one AR-overlay, transmitting the at least one AR overlay to at least one AR-client remote from the augmented reality device, merging, by the at least one AR-client, the at least one AR-overlay with a picture received from the at least one camera to provide an AR-image, and displaying the AR-image to a user of the at least one AR-client.FiguresFigure 1 is a schematic block diagram illustrating embodiments of an augmented reality system.Figure 2 is a schematic block diagram illustrating embodiments of displaying an AR-image.Figure 3 is a flowchart depicting embodiments of a method performed by an augmented reality system.Figure 4 is a schematic diagram illustrating a method to estimate heading/azimuth.Figure 5 is a flowchart depicting embodiments of adjusting heading.Detailed description of the inventionIt should be noted that the following embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present in another embodiment and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments. Figure 1 illustrates an embodiment of an augmented reality (AR) system 100. The AR-system comprises a global navigation satellite system module 101, an orientation measurement module 102, an augmented reality module 103, and at least one AR-client 104a, 104b.The at least one AR-client 104a, 104b comprises a camera 1040a, 1040b and a display 1041a, 1041b. The display 1041a, 1041b may be any kind of a display suitable for displaying an image to a user, such as a head-mounted display (HMD), and optical head mounted display, head-up display (HUD) or any kind of computer display. The computer display may be based on CRT, LCD, LCos, OLED and any other display technology. The camera 1040a, 1040b may be any kind of camera suitable for capturing images of the real world. The camera 1040a, 1040b may comprise both a night time camera and a day time camera.The global navigation satellite system (GNSS) module 101 is adapted to output position data of a carrier of the AR-system 100 to the at least on AR-client 104a, 104b. The GNSS module 101 may be any kind of navigational satellite system such as Global Positioning System (GPS) or Globalnaja navigatsionnaja sputnikovaja Sistema (GLONASS). The carrier of the AR-system 100 may be a person, a land vehicle, a watercraft or aircraft.The orientation measurement module 102 is adapted to output orientation data of the carrier of the AR-system 101 to the at least on AR-client 104a, 104b. The orientation data may be provided by an inertial measurement unit (IMU) that measures linear acceleration and rotational rate of the carrier. The IMU usually contains three gyros and three accelerometers. The IMU of the present invention may be a MEMS- (Microelectromechanical systems) device. MEMS-devices may be provided at low cost.The augmented reality module 103 is adapted to determine a position and orientation of the camera 1040a, 1040b of the at least one AR-client 104a, 104b based on the position data received from the GNSS-module 101 and orientation data received from the orientation measurement module 102. The augmented reality module 103 may be preconfigured with the position of the and orientation of the camera 1040a, 1040b relative to the GNSS-module 101 and the orientation measurement module 102 in order to determine the position and orientation of the camera 1040a, 1040b. Also referring to Figure 2, The AR-module 103 is then calculating screen positions of at least one AR object 201a, 201b based on the position and orientation of the camera of the at least one AR-client 104a, 104b to create at least one AR-overlay 200. The AR object 201a, 201b may be provided from an application specific information provider such as a battle management system or an architectural application. After the AR-module 103 has created the at least one AR-overlay 200, the AR-module 103 is transmitting the at least one AR overlay 200 to at least one AR-client 104a, 104b. The AR-module 103 may be implemented in a computer having a processor, a memory and wireless and/or wired communication modules.The at least one AR-client 104a, 104b is adapted to merging the at least one AR-overlay 200 with a picture 203 received from the camera 1040a, 1040b of the at least one AR-client 104a, 104b to provide an AR-image 204, and displaying the AR-image 204 on the display 1041a, 1041b. The AR-client 104a, 104b may be a remote weapon station, a mounted binocular, an operating room of a land vehicle, a watercraft or aircraft etc.The at least one AR-client 104a, 104b may also be adapted to merging the at least one AR-overlay 200 with a picture received from the camera of another AR-client. E.g. a picture of camera 1040a may be displayed with the AR-overlay 200 on display 1041b. In an alternative embodiment, the AR-module 103 may comprise a camera 1030, in the following the AR-module camera 1030. The AR-module camera 1030 may be any kind of camera suitable for capturing images of the real world. AR-module camera 1030 may comprise both a night time camera and a day time camera. The picture received from the AR-module camera 1030 may be transmitted to the AR-clients 104a, 104b with the AR-overlay 200 to be merged with the AR-client 104a, 104b and displayed on the display 1041a, 1041b.The AR-overlay 200 may be, and not limited to the following alternatives, in the form of an image with transparent background, data that tells how the ARoverlay shall be rebuilt by the AR-client, finished AR-image built by merging AR-symbols on top of the pictures from the AR-module camera, or finished AR-image built by merging AR-symbols on top of an image received by the AR-software from the AR-client (or any other available image source).When the augmented reality module 103 further comprises an AR-module camera 1030, the augmented reality module may be adapted to adjusting, based on the position data and orientation data a heading of the camera of the at least one AR-client by identifying earth fixed features captured by the camera of the augmented reality module 1030, tracking movement of the earth fixed features and adjusting the heading of the camera of the at least one AR-client by compensating for the movement of the earth fixed features.Embodiments of a method of displaying an augmented reality image will now be described with reference to Figure 3 and with continued reference to Figure 1 and 2.In step 301 navigational devices are read to find a position and orientation of an augmented reality device.The method comprises the steps of determining a position of an augmented reality device 103 by a global navigation satellite system and measuring an orientation of the augmented reality device by an orientation measurement module. In one embodiment, the method comprises determining the heading of the augmented reality device 103 by calculating change in position of the augmented reality device 103 to determine a first horizontal acceleration vector relative to north, calculating change in orientation of the augmented reality device to 103 determine a second horizontal acceleration vector, and determining the heading of the augmented reality device 103 by comparing the first horizontal acceleration vector with the second horizontal acceleration vector. An example is shown in Figure 4a – 4c where the method utilizes a GNSS receiver 101 (e.g. Global Positioning System (GPS)) and an inertial measurement unit 102 (IMU, containing three gyros and three accelerometers). The position is found by the GNSS. The roll and pitch are found by utilizing the direction of the gravity vector output from the IMU. The heading/azimuth may be estimated by utilizing the acceleration of the augmented reality device when in movement. The direction of the acceleration relative to the north can be found from the GNSS measurements. Figure 4a shows the acceleration is in the north direction as measured by the GNSS. Figure 4b shows a 2D example of how heading may be found from the acceleration. In real life the direction of the acceleration vector relative to the augmented reality device is measured by three accelerometers of the IMU measuring the direction and magnitude of the acceleration in 3D. When the same vector is known both relative to north and relative to the augmented reality device, the heading may be found. Figure 4b shows a general vehicle 401 in grey. Two accelerometers 402, 403 are attached to the vehicle 401, and accelerometer x 402 points in the vehicle forward direction, while accelerometer y 403 points to vehicle starboard. A horizontal acceleration is present, and for simplicity of the example both accelerometers measure the same value. Hence, the acceleration must point in a direction 45° relative to the vehicle, shown by the arrow 404. The same acceleration is visible in the GNSS measurements of Figure 4a, shown by arrow 405. When the two arrows 404 and 405 is overlaid 406 as shown in Figure 4c it is clear that the heading of the vehicle 401 is northwest.In step 302, the method is determining, based on the position and orientation of the augmented reality device 103, the position and orientation of at least one camera 1030, 1040a, 1040b. The camera may be any kind of camera suitable for capturing images of the real world, where the position and orientation of the camera relative to the augmented reality device is known. The camera may in exemplary embodiments be part of an AR-client 104a, 104b and/or part of the augmented reality device 103. The camera may comprise both a night time camera and a day time camera.In step 303, the method is calculating screen positions of at least one AR object 201a, 201b based on the position and orientation of the at least one camera to in step 304 create at least one AR-overlay 200. The step of calculating screen positions of the at least one AR object 201a, 201b to create at least one AR-overlay 200 may also take into consideration a state of the AR-client 104a, 104b obtained in step 308, such as day time camera, night time camera, camera field of view, amount camera zoom. The AR object 201a, 201b may be provided from an application specific information provider such as a battle management system or an architectural application depending on the configuration. The augmented reality device 103 may also calculate screen positions of the at least one AR object 201a, 201b based on the position and orientation of at least two cameras 1030, 1040a, 1040b to create a first AR-overlay based on the position and orientation of the first camera (e.g.1030), and a second AR-overlay based on the position of the second camera (e.g.1040a).In one embodiment, the first AR-overlay and the second AR-overlay contains the same at least one AR object (e.g.201a). In another embodiment, the first AR-overlay contains at least one AR object different from the second AR-overlay, e.g. the first AR-overlay may contain AR object 201a while the second AR-overlay contains AR object 201b. This allows producing completely independent AR-overlays to different AR-clients where independent inputs from AR-clients where the AR-clients may have their own cameras and where the directions of these cameras are inferred from the direction of the invention are utilised. Cameras of a weapon station, cameras mounted on a turret, and 360 degrees’ cameras would normally have well known positions relative to the augmented reality device such that the augmented reality device may calculate the direction of these cameras to provide AR-overlays to them. When the at least one AR-overlay 200 has been created, then in step 305, the method is transmitting the at least one AR overlay 200 to at least one AR-client 104a, 104b remote from the augmented reality device 103.In step 306, the method is merging, by the at least one AR-client 104a, 104b, the at least one AR-overlay 200 with a picture received from the at least one camera 1030, 1040a, 1040b to provide an AR-image, and in step 307 displaying the AR-image to a user of the at least one AR-client 104a, 104b. The AR-image may be displayed to a user using any kind of a display 1041a, 1041b suitable for displaying an image to the user, such as a head-mounted display (HMD), and optical head mounted display, head-up display (HUD) or any kind of computer display. In one embodiment, a first AR-client merges the first AR-overlay with a picture received from the first camera to provide a first AR-image, and a second AR-client merges the second AR-overlay with a picture received from the second camera to provide a second AR-image.When the AR-device 103 is stationary the estimate of the heading may drift, in particular, when using small, low cost IMUs 102. To position the AR objects 201a, 201b with the required accuracy it is necessary to adjust for this drift. In one embodiment, when the AR device 103 is provided with an AR-module camera 1030 then, with reference to Figure 5, in step 302 the method further comprises adjusting the heading of the augmented reality device 103 by in step 501 identifying earth fixed features captured by the AR-module camera 1030 of the augmented reality device 103, in step 502 tracking movement of the earth fixed features, and in step 503 adjusting the heading of the augmented reality device 103 by compensating for the movement of the earth fixed features. The identification and tracking of earth fixed features may be performed with any suitable video tracking software.When using the word "comprise" or “comprising” it shall be interpreted as nonlimiting, i.e. meaning "consist at least of".Modifications and other embodiments of the disclosed embodiments will come to mind to one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the embodiment(s) is/are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of this disclosure. Although specific terms may be employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation. Therefore, the above embodiments should not be taken as limiting the scope, which is defined by the appending claims.Also note that terminology such as a first computer and a second computer should be considered to be non-limiting and does in particular not imply a certain hierarchical relation between the two.
Claims
Claims1. Augmented reality (AR) system, comprising,- a global navigation satellite system module adapted to output position data, - an orientation measurement module adapted to output orientation data, - an augmented reality module,- at least one AR-client comprising a camera and a display,whereinthe augmented reality module is adapted to:- determine a position and orientation of the camera of the at least one AR-client based on the position data and orientation data,- calculating screen positions of at least one AR object based on the position and orientation of the camera of the at least one AR-client to create at least one AR-overlay,- transmitting the at least one AR overlay to at least one AR-client, and the AR-client is adapted to:- merging the at least one AR-overlay with a picture received from the camera of the at least one AR-client to provide an AR-image, and-displaying the AR-image on the display.2. Augmented reality system according to claim 1, wherein the augmented reality module further comprises a camera.3. Augmented reality system according to claim 2, wherein the augmented reality module is further adapted to adjusting, based on the position data and orientation data a heading of the camera of the at least one AR-client by:- identifying earth fixed features captured by the camera of the augmented reality module,- tracking movement of the earth fixed features,- adjusting the heading of the camera of the at least one AR-client by compensating for the movement of the earth fixed features.4. Augmented reality system according to claim 1, wherein the orientation measurement module comprises an inertial measurement unit.5. Augmented reality system according to claim 4, wherein the inertial measurement unit is a MEMS-device.6. Augmented reality system according to claim 1 or 2, wherein the camera of the at least one AR-client and/or the camera of the augmented reality module comprises at least a night time camera and a daytime camera.7. Method of displaying an augmented reality (AR) image, comprising, -determining a position of an augmented reality device by a global navigation satellite system,-measuring an orientation of the augmented reality device by an orientation measurement module,-determining, based on the position and orientation of the augmented reality device, the position and orientation of at least one camera,-calculating screen positions of at least one AR object based on the position and orientation of the at least one camera to create at least one AR-overlay, -transmitting the at least one AR overlay to at least one AR-client remote from the augmented reality device,-merging, by the at least one AR-client, the at least one AR-overlay with a picture received from the at least one camera to provide an AR-image,-displaying the AR-image to a user of the at least one AR-client.8. Method according to claim 7, further comprises calculating screen positions of the at least one AR object based on the position and orientation of at least two cameras to create a first AR-overlay based on the position and orientation of the first camera, and a second AR-overlay based on the position of the second camera.9. Method according to claim 8, wherein a first AR-client merges the first AR-overlay with a picture received from the first camera to provide a first AR-image, and a second AR-client merges the second AR-overlay with a picture received from the second camera to provide a second AR-image.10. Method according to claim 8, wherein the first AR-overlay and the second AR-overlay contains the same at least one AR object.11. Method according to claim 8, wherein the first AR-overlay contains at least one AR object different from the second AR-overlay.12. Method according to claim 7, wherein the at least one AR-object is received from an information provider.13. Method according to claim 12, wherein the information provider is a battlefield management system.14. Method according to claim 7, wherein determining, based on the position and orientation of the augmented reality device, the position and orientation of at least one camera, comprises determining the heading of the augmented reality device by:-calculating change in position of the augmented reality device to determine a first horizontal acceleration vector relative to north,-calculating change in orientation of the augmented reality device to determine a second horizontal acceleration vector, and-determining the heading of the augmented reality device by comparing the first horizontal acceleration vector with the second horizontal acceleration vector.15. Method according to claim 7, wherein determining, based on the position and orientation of the augmented reality device, the position and orientation of at least one camera, comprises adjusting the heading of the augmented reality device by: - identifying earth fixed features captured by a camera of the augmented reality device,- tracking movement of the earth fixed features,- adjusting the heading of the augmented reality device by compensating for the movement of the earth fixed features.Patentkrav1. Utvidet virkelighet(AR)-system, omfattende- en global navigasjonssatellittsystemmodul tilpasset å mate ut posisjonsdata, - en orienteringsmålingsmodul tilpasset å mate ut orienteringsdata,- en utvidet virkelighet-modul,- minst én AR-klient omfattende et kamera og et display,hvoriutvidet virkelighet-modulen er tilpasset å:- bestemme en posisjon og orientering av den minste ene AR-klientens kamera basert på posisjonsdataen og orienteringsdataen,- beregne skjermposisjoner for minst ett AR-objekt basert på posisjonen og orienteringen av den minst ene AR-klientens kamera for å danne minst ett AR-lag,- sende det minst ene AR-laget til den minst ene AR-klienten, ogAR-klienten er tilpasset å:- kombinere det minst ene AR-laget med et bilde mottatt fra den minst ene AR-klientens kamera for å tilveiebringe at AR-bilde, og- fremvise AR-bildet på displayet.2. Utvidet virkelighet-system ifølge krav 1, hvori utvidet virkelighet-modulen ytterligere omfatter et kamera.3. Utvidet virkelighet-system ifølge krav 2, hvori utvidet virkelig-modulen ytterligere er tilpasset å justere, basert på posisjonsdataen og orienteringsdataen en kurs av den minst ene AR-klientens kamera ved å:- identifisere bakkestasjonære trekk fanget av utvidet virkelighet-modulens kamera,- spore bevegelse av de bakkestasjonære trekk,- justere kursen av den minst ene AR-klientens kamera ved å kompensere for bevegelsen av de bakkestasjonære trekk.4. Utvidet virkelighet-system ifølge krav 1, hvori orienteringsmålingsmodulen omfatter en treghetsmålingsenhet.5. Utvidet virkelighet-system ifølge krav 4, hvori treghetsmålingsenheten er en MEMS-innretning.6. Utvidet virkelighet-system ifølge krav 1 eller 2, hvori den minst ene AR-klientens kamera og/eller utvidet virkelighet-modulens kamera omfatter minst ett nattkamera og et dagkamera.7. Fremgangsmåte for å fremvise et utvidet virkelighet(AR)-bilde, omfattende, - bestemme en posisjon av en utvidet virkelighet-innretning med et globalt navigasjonssatellittsystem,- måle en orientering av utvidet virkelighet-innretningen med en orienteringsmålingsmodul,- bestemme, basert på posisjonen og orienteringen av utvidet virkelighetinnretningen, posisjonen og orienteringen av minst ett kamera,- beregne skjermposisjoner for minst ett AR-objekt basert på posisjonen og orienteringen av det minst ene kameraet for å danne minst ett AR-lag,- sende det minst ene AR-laget til minst én AR-klient fjernt fra utvidet virkelighetinnretningen,- kombinere, med den minst ene AR-klienten, det minst ene AR-laget med et bilde mottatt fra det minst ene kameraet for å tilveiebringe at AR-bilde, og - fremvise AR-bildet til en bruker av den minst ene AR-klienten.8. Fremgangsmåte ifølge krav 7, ytterligere omfattende å beregne skjermposisjoner for det minst ene AR-objektet basert på posisjonen og orienteringen av minst to kamera for å danne et første AR-lag basert på posisjonen og orienteringen av det første kameraet, og et andre AR-lag basert på posisjonen av det andre kameraet.9. Fremgangsmåte ifølge krav 8, hvori en første AR-klient kombinerer det første AR-laget med et bilde mottatt fra det første kameraet for å tilveiebringe et første AR-bilde, og en andre AR-klient kombinerer det andre AR-laget med et bilde mottatt fra det andre kameraet for å tilveiebringe et andre AR-bilde.10. Fremgangsmåte ifølge krav 8, hvori det første AR-laget og det andre AR-laget inneholder det samme minst ene AR-objektet.11. Fremgangsmåte ifølge krav 8, hvori det første AR-laget inneholder minst ett AR-objekt forskjellig fra det andre AR-laget.12. Fremgangsmåte ifølge krav 7, hvori det minst ene AR-objektet mottas fra en informasjonstilbyder.13. Fremgangsmåte ifølge krav 12, hvori informasjonstilbyderen er et stridsledelsessystem.14. Fremgangsmåte ifølge krav 7, hvori å bestemme, basert på posisjonen og orienteringen av utvidet virkelighet-innretningen, posisjonen og orienteringen av det minste ene kameraet, omfatter å bestemme utvidet virkelighet-innretningens kurs ved å:- beregne endring i posisjon av utvidet virkelighet-innretningen for å bestemme en første horisontal akselerasjonsvektor relativt til nord,-beregne endring i orientering av utvidet virkelighet-innretningen for å bestemme en andre horisontal akselerasjonsvektor, og-bestemme utvidet virkelighet-innretningens kurs ved å sammenligne den første horisontale akselerasjonsvektoren med den andre horisontale akselerasjonsvektoren.15. Fremgangsmåte ifølge krav 1, hvori å bestemme, basert på posisjonen og orienteringen av utvidet virkelighet-innretningen, posisjonen og orienteringen av minst ett kamera, omfatter å justere utvidet virkelighet-innretningens kurs ved å: - identifisere bakkestasjonære trekk fanget av et utvidet virkelighet-modulkamera, - spore bevegelse av de bakkestasjonære trekk,- justere utvidet virkelighet-innretningens kurs ved å kompensere for bevegelsen av de bakkestasjonære trekk.
Drawings

Priority


No priority

Patent classes


IPC classesG06T 19/00G06F 3/14G06T 7/70H04N 7/18CPC classesG06T 19/006G06F 3/14G09G 2354/00G06T 7/70H04N 7/181

Citations


GB 2477787 ()EP 1435737 ()WO 2012049674 ()US 2014079281 ()

Applicants, owners and inventors


Owners
Augmenti AS
Innherredsveien 7 A 7014 TRONDHEIM NO (TRONDHEIM Municipality, Trøndelag county)Org. number: 992104678
Inventors
Kenneth Gade
Prost Stabels vei 453 2019 SKEDSMOKORSET NO (LILLESTRØM Municipality, Akershus county)
Geir Meyer
Danseberget 35 1792 TISTEDAL NO (HALDEN Municipality, Østfold county)
Lars Inge Solhaug
Ranheimsfjæra 125 7055 RANHEIM NO (TRONDHEIM Municipality, Trøndelag county)

Agent


Valea AB
Box 1098 40523 GÖTEBORG SE
Org. number:
Reference: PG20918NO00

Status history


Payment history


Oppositions and appeals


InnsigelseReceived: 06.05.2019 NIPO's case No.: 2019/00190Current status: Avgjort. Innsigelsen har ført fram
OpponentHÅMSØ PATENTBYRÅ AS
HÅMSØ PATENTBYRÅ AS

Org. number: 918752900

Name and address are automatically synchronized with the Brønnøysund Register Centre.

Open in The Brønnøysund Register Centre
Postboks 9 4068 STAVANGER NO

Administrative limitation


No administrative limitation

Liens and licences


No liens or licenses registered

Gazette entries


No Gazette entries foundOnly events from 2024.03.04 and onwards are shown.