Key information

Title: Coherent Node Controller
Application number: 20171434
Case type: National
Status: 02.03.2020 Patent meddelt (B1)
Filed: 05.09.2017
Effective date: 05.09.2017
Publicly available: 06.03.2019
Next annual fee due: 30.09.2027
Applicant: Numascale AS (NO)
Owner: Numascale AS (NO)
Inventor: Thibaut Palfer-Sollier (NO) et al.
Agent: BRYN AARFLOT AS (NO)
Granted: 02.03.2020
Patent number: 344681
Expiry date: 05.09.2037

Abstract and drawing

Disclaimer: This text has been machine-scanned and may contain errors – please refer to "Publications" for legally binding content.

The invention relates to architecture for multiprocessors and more particularly to a multiport node controller. The a cache coherent node controller at least comprising: one or more network interface controllers, each network interface controller includes at least one network interface, and at least two coherent interfaces each configured for communication with a microprocessor.

Publications

Checking availability of the latest published version...

Documents

Expand

Download files (26)

Date

Date	Doc. No.	Process	Case number	In/Out	Journal description	To/from
05.03.2020	15-01	Saksbehandling	20171434	UT	PT Registreringsbrev Nasjonal Patent (15) (PT20171434)
18.12.2019	14-01	Saksbehandling	20171434	UT	Intention to grant	BRYN AARFLOT AS
18.12.2019	13-01	Saksbehandling	20171434	INN	Patenttegninger
18.12.2019	12-01	Saksbehandling	20171434	INN	Engelske krav
06.08.2019	11-01	Saksbehandling	20171434	UT	PT Varsel om betaling av første årsavgift (3317) (PT20171434)
20.05.2019	10-05	Saksbehandling	20171434	INN	Beskrivelse merket	BRYN AARFLOT AS
20.05.2019	10-04	Saksbehandling	20171434	INN	Hovedbrev	BRYN AARFLOT AS
20.05.2019	10-03	Saksbehandling	20171434	INN	Krav	BRYN AARFLOT AS
20.05.2019	10-02	Saksbehandling	20171434	INN	Beskrivelse ren	BRYN AARFLOT AS
20.05.2019	10-01	Saksbehandling	20171434	INN	Korrespondanse (Hovedbrev inn)	BRYN AARFLOT AS
20.03.2019	09-01	Saksbehandling	20171434	INN	Korrespondanse (Hovedbrev inn)	BRYN AARFLOT AS
17.01.2019	08-01	Saksbehandling	20171434	INN	Korrespondanse (Hovedbrev inn)	BRYN AARFLOT AS
20.10.2018	07-01	Saksbehandling	20171434	UT	Realitet patent	BRYN AARFLOT AS
29.05.2018	06-04	Saksbehandling	20171434	INN	Beskrivelse ren	BRYN AARFLOT AS
29.05.2018	06-03	Saksbehandling	20171434	INN	Beskrivelse merket	BRYN AARFLOT AS
29.05.2018	06-02	Saksbehandling	20171434	INN	Hovedbrev	BRYN AARFLOT AS
29.05.2018	06-01	Saksbehandling	20171434	INN	Korrespondanse (Hovedbrev inn)	BRYN AARFLOT AS
28.02.2018	05-02	Saksbehandling	20171434	UT	PT report 03:58:42
28.02.2018	05-01	Saksbehandling	20171434	UT	Realitet patent	BRYN AARFLOT AS
26.09.2017	04-03	Saksbehandling	20171434	INN	Fullmakt	Bryn Aarflot AS
26.09.2017	04-02	Saksbehandling	20171434	INN	Erklæring Søkers rett til oppfinnelse	Bryn Aarflot AS
26.09.2017	04-01	Saksbehandling	20171434	INN	Korrespondanse (Hovedbrev inn)	Bryn Aarflot AS
08.09.2017	03-01	Saksbehandling	20171434	UT	Formal Examination 1	Bryn Aarflot AS
08.09.2017	02-01	Saksbehandling	20171434	UT	Infobrev til oppfinner	Bryn Aarflot AS
05.09.2017	01-02	Saksbehandling	20171434	INN	Beskrivelse	Bryn Aarflot AS
05.09.2017	01-01	Saksbehandling	20171434	INN	Søknadsskjema Patent	Bryn Aarflot AS

Description, claims and drawings

Disclaimer: This text has been machine-scanned and may contain errors – please refer to "Publications" for legally binding content.

Description

Technical Field

[0001] The present invention relates to architecture for multiprocessors and more particularly to a cache coherent node controller.

Background Art

[0002] Modern processors have capabilities of interconnecting a relatively low number of processor modules to form larger systems by direct connections between the processors across point-to-point links. These links are not capable of expanding the system size beyond typically 4-8 modules, although some may go as far as a maximum of 16. Each processor module contains several CPU cores that may be integrated on a single die or a combination of two or more dies in a multi-chip module. For performance reasons, the directly connected systems are limited to 8 modules due to the increasing overhead due to the task of maintaining data coherency across all memories and high speed caches in the system.

[0003] Modern high performance computing (HPC) systems normally use nodes with 2-CPU sockets that are connected to an interconnect switch through a network interface controller (NIC).

[0004] The present invention is designed to solve several of the most pressing problems with very large scale computer systems. One of the major problems is the amount of energy that it takes to drive inter-node signals through an interconnect switching system. Another is the number of operating system instances that are needed to handle the resources contained in each node. Reducing this number will contribute to better efficiency and utilization of resources and it will also be more flexible for running programs of various sizes and complexities.

[0005] Background art is found in US 2008/0270708 A1, Warner et al. titled System And Method for Achieving Cache Coherency Within Multiprocessor Computer System, further reference is made to US 6,092,155 (Olnowich) which discloses a cache coherent network adapter for scalable shared memory systems and US 2014/0201326 A1 (Joshua et al.), which discloses an interconnected ring network in a multi-processor system.

Disclosure of Invention

[0006] It is one object of the invention to provide systems and methods that do not suffer from the problems indicated above. By introducing a multi-ported node controller, the number of nodes in a large system can be reduced in proportion with the number of coherent ports on the node controller. The processor modules that are connected to the node controller will communicate through on-chip parallel data paths and thereby increasing throughput, reducing latency and reducing power consumption by eliminating the need to drive external wires for the communication between the processor modules in the node.

[0007] This and other objects are met by a computer system at least comprising two CNCNICs and a network switch.

[0008] In a computer system multi-port interconnect modules may contain multiple instances of protocol transaction processing engines that perform the necessary transactions on directories and tags to maintain the system wide cache and memory coherency.

[0009] More particularly according to one embodiment it is provided a cache coherent node controller at least comprising:

a.one or more network interface controllers, each network interface controller includes at least one network interface, and

b.at least two coherent interfaces each configured for communication with a microprocessor. The cache coherent controller may further comprise: c. one or more interconnected protocol engines,

d.one or more coherent link layers, each coherent link layer at least connected to one protocol engine, each coherent link layer provides coherent interfaces.

[0010] According to another embodiment it is provided a computer system comprising one or more nodes where each node is connected to at least one network switch, each node at least comprising:

e.a cache coherent node controller at least comprising:

i. one or more network interface controllers, each network interface controller includes at least one network interface, and

ii. at least two coherent interfaces each connected to a microprocessor.

[0011] The cache coherent node controller of the computer system may further comprise:

iii. one or more interconnected protocol engines,

iv. one or more coherent link layers, each coherent link layer at least connected to one protocol engine, each coherent link layer provides coherent interfaces.

[0012] Connection between each node and the network switches can be provided by the network interfaces.

[0013] According to one aspect of the system one network interface of each cache coherent node controller can be connected to one network switch to form a 1-dimensional fat tree topology.

[0014] In another aspect two network interfaces of each cache coherent node controller can be connected to two network switches to form a 2-dimensional fat tree topology.

[0015] In yet an aspect of the invention three network interfaces of each cache coherent node controller can be connected to three network switches to form a 3-dimensional fat tree topology.

[0016] In yet an aspect four network interfaces of each cache coherent node controller can be connected to four network switches to form a 4-dimensional fat tree topology.

[0017] In yet an aspect n network interfaces of each cache coherent node controller can be connected to n network switches to form an ndimensional fat tree topology.

[0018] A system according to claim 5where the network switches are one of: an Omnipath switch, an InfiniBand switch, a Gen-Z switch or an Ethernet switch.

[0019] A system according to claim 5 where connection between each node and the network switches is provided by fibre optic links to the network switches.

[0020] Other advantageous features will be apparent from the accompanying claims.

Brief Description of Drawings

[0021] Following is a brief description of the drawings in order to make the invention more readily understandable, the discussion that follows will refer to the accompanying drawings, in which

[0022] Fig. 1 shows a simplified drawing of an 8-port cache coherent node controller with 4 integrated network interfaces. Configurations require one or more cache coherent ports and one or more network interfaces,

[0023] Fig. 2 shows 8 cache coherent links to CPUs and 4 network interfaces (NICs) to connect to HPC switches with various number of ports. The 4 NICs allow configurations with up to 4 dimensional fat trees,

[0024] Fig. 3 shows CPU connections,

[0025] Fig. 4 shows a simplified drawing of an 8-port cache coherent node controller with 4 integrated network interfaces and 4 connected CPUs. Configurations require one or more cache coherent ports and one or more network interfaces. Figure 5, 7, 8a and 8b, use a simplified ring depicting a node with two or more CPUs and two or more NIC channels,

[0026] Fig. 5 shows a 2-D fat tree system with 32 nodes,

[0027] Fig. 6 shows a 1-D fat tree, with 8-node configurations, where top: 8 nodes is associated with one 8-port node controller and bottom 8 nodes are associated with two 8-port node controllers for double bandwidth and redundancy,

[0028] Fig. 7 shows a 2-D fat tree system with 64 nodes,

[0029] Fig. 8a shows a 2D face of with 48 nodes of a 3-D fat tree system,

[0030] Fig. 8b shows a 3-D fat tree system with 6 X 8 X 6 nodes,

[0031] Fig. 9 shows a fat tree configuration in a rack, where a node includes 8 cache coherent, a rack contains 32 nodes, 8 x 64 port Switches, Message Passing Total: 256CPU’s, and

[0032] Fig. 10 shows a simplified drawing of an 8-port cache coherent node controller with 4 integrated network interfaces. Configurations require one or more cache coherent ports and one or more network interfaces. Further drawing use a simplified ring depicting a node with two or more CPUs and two or more NIC channels.

Detailed description of the Invention

[0033] In the following it is firstly disclosed general embodiments in accordance to the present invention, thereafter particular exemplary embodiments will be described. Where possible reference will be made to the accompanying drawings and where possible using reference numerals in the drawings. It shall be noted however that the drawings are exemplary embodiments only and other features and embodiments may well be within the scope of the invention as described.

[0034] In the context of the present invention a processor is defined as a module containing one or more CPU cores, one or more memory channels, one or more I/O channels, one or more coherent links and memory in the form caches or with or without main memory in any form included in the processor module.

[0035] A node contains one or more processor modules, one or more memory banks, zero or one or more I/O modules and one or more coherent node controllers.

[0036] In the present invention it is not necessarily different bandwidths within a fat tree configuration as indicated in the figures.

[0037] The present invention is based on integrating several functions that in combination will produce a flexible and versatile interconnect component for building large scale computer systems. The key of the invention is the close connection between a packet-based network switch and a cache coherent switch for inter-processor communication inside the node. This connection enables building of scalable systems with low latency and high bandwidth communication with very low power consumption. As an example, a device with 8 coherent links to CPUs and 4 interconnect links to connect to other devices of the same kind will consume less than 100W of power. Such a device will reduce the main system interconnect port requirement with a factor of four compared to the traditional configuration for HPC systems with 2 CPUs per network interface.

[0038] A Network Interface Controller, NIC, can be implemented for use with different types of high performance networks like Ethernet, InfiniBand, Omnipath (Intel), BXI (ATOS/Bull), and Gen-Z (Gen-Z consortium standard). Gen-Z defines a load/store architecture where CPU instructions can address memory directly without going through a message passing protocol through software driver layers. Omnipath has a similar function in addition to RDMA functionality for message passing. The ability for processors to use load and store instructions to operate directly on memory locations across the interconnect reduces the overhead for data movement. This is most important for short data transfers where latency and overhead becomes a significant factor for performance.

First embodiment of the present invention

[0039] The present invention will now be described with a specified example. The invention introduces a new element, namely a cache coherent node controller, it also provides examples of the use of said new element in system configurations from 1-dimesnional fat tree configurations to ndimensional fat tree configurations.

[0040] The cache coherent node controller may at least comprise:

a. one or more network interface controllers, each network interface controller includes at least one network interface. An example of such a coherent node controller is shown schematically in figure 1 and more detailed in figure 2. It can be seen in figure 2 that the node controller includes 4 NICs and each NIC includes at least one network interface port. The NICs are configured for communication with network switches.

b. at least two coherent interfaces each configured for communication with a microprocessor. In figure 2 it is shown eight coherent interfaces, CLL. Each of the CLLs at least comprises a coherent interface port.

c. one or more interconnected protocol engines. The protocol engines are shown in figure 2 as squares marked PE. Figure 2 indicates 16 Pes. Each PE is connected directly or indirectly to a CLL. PEs are interconnected directly or indirectly.

[0041] The cache coherent node controller can be scaled up or down maintaining the same principle of incorporating one or more NICs.

[0042] Obviously, the design layout shown in figure 2 with NICs arranged in the corner is a design matter, also it makes the figure readily understandable.

Second embodiment of the present invention

[0043] In the second embodiment of the present invention the coherent node controller with NICs described above is used in a computer system. The computer system comprises one or more nodes where each node is connected to at least one network switch, fig.5, 7.8a, 8b and 9. Each node is described in the section above, the first embodiment of the present invention.

[0044] In the event that only one NIC is included or only one NIC is connected to network switches one will have a one dimensional system as depicted in figure 6. In figure 5 it is shown a two-dimensional system, i.e. two NICs in each cache coherent node controller is connected to a network switch.

[0045] Figure 7 shows a three dimensional system, that is; each NIC is connected with three networks switches. Fig 8a shows a front layer of a three dimensional system whilst figure 8b shows a three dimensional system with all layers. The configuration in fig.8b is 6 x 8 x 6 nodes.

[0046] The cache coherent node controller of fig 2, is configured for four dimensional systems.

[0047]

Claims

1. A cache coherent node controller at least comprising:

a. one or more network interface controllers, each network interface controller includes at least one network interface, and

b. at least two coherent interfaces each configured for communication with a microprocessor.

2. A cache coherent node controller according to claim 1, where the cache coherent controller further comprises:

c. one or more interconnected protocol engines,

d. one or more coherent link layers, each coherent link layer at least connected to one protocol engine, each coherent link layer provides coherent interfaces.

3. A computer system comprising one or more nodes where each node is connected to at least one network switch, each node at least comprising: a. a cache coherent node controller at least comprising:

i. one or more network interface controllers, each network interface controller includes at least one network interface, and

ii. at least two coherent interfaces each connected to a microprocessor.

4. A computer system according to claim 3, where the cache coherent node controller further comprises:

iii. one or more interconnected protocol engines,

iv. one or more coherent link layers, each coherent link layer at least connected to one protocol engine, each coherent link layer provides coherent interfaces.

5. A computer system according to claim 3 or 4, where connection between each node and the network switches is provided by the network interfaces.

6. A system according to any of the claims 3 – 5 where one network interface of each cache coherent node controller is connected to one network switch to form a 1-dimensional fat tree topology.

7. A system according to any of the claims 3 – 5 where two network interfaces of each cache coherent node controller is connected to two network switches to form a 2- dimensional fat tree topology.

8. A system according to any of the claims 3 – 5 where three network interfaces of each cache coherent node controller is connected to three network switches to form a 3- dimensional fat tree topology.

9. A system according to any of the claims 3 – 5 where four network interfaces of each cache coherent node controller is connected to four network switches to form a 4- dimensional fat tree topology.

10.A system according to any of the claims 3 – 5 where n network interfaces of each cache coherent node controller is connected to n network switches to form a n- dimensional fat tree topology.

11.A system according to claim 5where the network switches are one of: an Omnipath switch, an InfiniBand switch, a Gen-Z switch or an Ethernet switch.

12.A system according to claim 5 where connection between each node and the network switches is provided by fibre optic links to the network switches.

Patentkrav

1. En cache-koherent node-kontroller i det minste omfattende:

a) en eller flere nettverksgrensesnittkontrollere, der hver nettverksgrensesnittkontroller i det minste inkludere ett nettverksgrensesnitt, og

b) minst to koherente grensesnitt som hver er konfigurert for kommunikasjon med en mikroprosessor.

2. En cache-koherent node-kontroller ifølge krav 1, der den cachekoherente kontrolleren videre omfatter:

c) en eller flere sammenkoblede protokollmotorer,

d) ett eller flere koherente koblingslag, der hvert koherente koblingslag i det minste er koblet til en protokollmotor, og der hvert koherente koblingslag tilveiebringer koherente grensesnitt.

3. Et datasystem omfattende en eller flere noder hvor hver node er koblet til minst en nettverksbryter, idet hver node i det minste omfatter:

a) en cache-koherent node-kontroller i det minste omfattende:

ii. en eller flere nettverksgrensesnittkontrollere, der hver nettverksgrensesnittkontroller i det minste inkluderer ett nettverksgrensesnitt, og

ii. minst to koherente grensesnitt som hver er koblet til en mikroprosessor.

4. Et datasystem ifølge krav 3, der den cachekoherente nodekontrolleren videre omfatter:

iii. en eller flere sammenkoblede protokollmotorer,

iv. ett eller flere koherente koblingslag, der hvert koherente koblingslag i det minste er koblet til en protokollmotor, og der hvert koherente koblingslag tilveiebringer koherente grensesnitt.

5. Et datasystem ifølge krav 3 eller 4, hvor forbindelsen mellom hver node og nettverksbryterne er tilveiebragt av nettverksgrensesnittene.

6. Et system ifølge et hvilket som helst av kravene 3 - 5 hvor ett nettverksgrensesnitt for hver cachekonherent node-regulator er koblet til en nettverksbryter for å danne en 1-dimensjonal fat tree topology.

7. Et system ifølge et hvilket som helst av kravene 3 - 5 hvor to nettverksgrensesnitt for hver cachekonherente node-kontroller er koblet til to nettverksbrytere for å danne en 2-dimensjonal fat tree topology.

8. Et system ifølge et hvilket som helst av kravene 3 - 5, hvor tre nettverksgrensesnitt av hver cache-koherente node-regulator er koblet til tre nettverksbrytere for å danne en tredimensjonal fat tree topology.

9. Et system ifølge et hvilket som helst av kravene 3 - 5 hvor fire nettverksgrensesnitt for hver cache-koherente node-regulator er koblet til fire nettverksbrytere for å danne en 4-dimensjonal fat tree topology.

10. Et system ifølge et hvilket som helst av kravene 3 - 5, hvor n nettverksgrensesnitt for hver cache-koherente node-kontroller er koblet til nnettverksbrytere for å danne en n-dimensjonal fat tree topology.

11. Et system ifølge krav 5 hvor nettverksbryterne er en av: en Omnipathbryter, en InfiniBand-bryter, en Gen-Z-bryter eller en Ethernet-bryter.

12. Et system ifølge krav 5, hvor forbindelsen mellom hver node og nettverksbryterne er tilveiebrakt ved fiberoptiske koblinger til nettverksbryterne.

Drawings

Priority

No priority

Patent classes

IPC classes

G06F 12/08

G06F 12/0802

CPC classes

G06F 12/08

G06F 12/0802

Citations

US 2008270708 ()

US 6092155 ()

US 2014201326 ()

Applicants, owners and inventors

Owners

Numascale AS

Innspurten 15

0663 OSLO