Sometimes a moment of lack calculation resources is occurs. Some tasks start to consume more time or even can't be completed.
Usual way to fix this is to use more powerful hardware: High-End desktops (HEDT) and servers.
Another way is to use clusters.
Cluster is a system of multiple computers which act as one entity.
Clusters are suitable when new powerful hardware is unavailable by financial or political issues.
Clusters are not so expensive. There are kind of clusters called Beowolf which are assemled from regular desktops. Also there are kind of clusters from microcomputers like Raspberry Pi.
Well known fact that each year desktops with multicore CPUs are become cheaper and cheaper.
Umvirt Linux From Scratch (ULFS) is contain software packages which can be used to create clusters offline.
All is needed to create cluster with ULFS are:
This site is describe how to build clusters with ULFS.
Clusters created with ULFS can be used to:
To build a cluster some software packages can be used:
Cluster contain major parts:
Also can be useful:
Storage access network (SAN) is local area network (LAN) which used by compute units to access virtual disk drives (disk images) on Network Access Storages (NAS).
Such network should have a good bandwith and should be isolated from other networks.
Network Access Storages (NAS) is servers which stored virtual drives (disk images) of compute units.
NASes is use next protocols:
Also other protocols can be used: LFS, SMB/CIFS, HTTP.
To boot compute unit from virtual drives (disk images) over storage access network (SAN) special network booting Preboot Execution Environment (PXE) server is needed.
Such server act as DHCP-server and perform initial PXE boot.
Compute units are desktops/servers with network boot.
Network boot allows perform disk cloning at high speed and reduce total cost of each compute unit.
Compute units can be used without HDD/SDD and VGA card. Modern CPUs (like AMD Ryzen 7xxx, 9xxx series) is have built-in integrated graphics no VGA card is needed.
On-boot diagnostics can be perfomed with beeper and serial port (COM/USB) console. Remote access can be performed by various KVM/IPKVM/IPMI solutions.
In addition to SAN compute units are connected to regular local area network which can be connected to other networks.
Cluster building is not trivial task. It should be completed step-by-step in order to fix errors on each stage.
In order to reduce costs first deployment can be done in virtual environment.
It's possible to use "qemu libvirt virt-manager" stack from ULFS.
In order to reduce a disk space consumption and cloning and deployment time ZFS, LVM, QCOW2 snapshots can be used instead of RAW disk images.
In theory QEMU package offers a network block device (NBD) interface for QCOW2 images which can be used with iSCSI protocol. But on practice this option is not works correctly with open-iscsi.
It's possible to use sepecialized storage solutions as PXE-server for ULFS powered compute nodes: