News
15 February 2021 Reading time: 8 minutes

Lev Korolkov

Head of Cloud Services at EFSOL Oblako

Ceph-based budget storage without buying data storage systems

Ceph-based budget storage without buying data storage systems

 

To provide traditional data storage, you need to buy a storage system. With Ceph distributed storage, you can store data on the computing side and reduce costs. Lev Korolkov, Head of Cloud Services at EFSOL Oblako has calculated how much Ceph can help save. This article is based on 12 years of experience in systems integration and business processes automation.

Contents

How distributed Ceph storage works

Ceph is a storage based on freeware. It allows you to merge disk space of servers. With Ceph you can arrange a scalable storage with high performance and redundancy. Ceph can be used to create object storage and as a block device.

In Serh, each block of data is replicated to multiple servers. This provides fault tolerance.

Ceph operating principle: data is broken down into blocks Each block is stored in two copies on different nodesх

The default replication factor in Ceph is two. If you are not using fault-tolerant disk arrays, I recommend setting a higher replication factor.

If one of the nodes fails, Ceph will record that the data block is unavailable. After a certain time (300 seconds by default) it will start to restore the missing data.

Restoring redundancy in Ceph. One of the nodes failed. Data copies that were stored on it are duplicated to other nodes

If a new server is installed in the cluster, the storage becomes unbalanced: disks on all nodes are evenly filled with data. In Ceph, this mechanism is called Crush.

To get high disk space performance in Ceph, use Cache Tiering - multi-level caching. A separate high-performance pool is used for caching, and the main data are placed on inexpensive hard drives.

Cache Tiering mechanism in Ceph: fast SSD disks are used for caching, and the main data is placed on the HDD

With Ceph, you can set up budget storage or a private cloud without using a data storage system. We calculated how much these solutions would cost.

Storage for medium and small businesses

ОThe task is to provide a fault-tolerant 6TB storage.

Storage without the use of Ceph

Without Ceph, you would need to purchase storage in the following configuration:

  • Fujitsu ETERNUS DX100 storage system;
  • 12 Nearline SAS 1Tb SFF (RAID10);

The solution will cost 19,000 USD.

The cost of a 6TB storage with purchase of a data storage system

ConfigurationIOPs (read 70%/write 30%, Block size 4K)
IOPs are calculated for created arrays of NL SAS drives in a storage system and SATA drives in Ceph. Caching was disabled because it leads to overstating of IOPs.
Useful capacity, GBPrice per 1 GB, USDTotal cost, USD
Fujitsu ETERNUS DX100 + 12 Nearline SAS 1Tb SFF (RAID10)7606 0003,219 000

Ceph-based storage

You can do without buying a storage system if you use Ceph. I suggest Supermicro Twin as a storage solution. These are four server platforms in a single two-unit high enclosure. Its main nodes are duplicated for greater fault tolerance.
The contents of each node:

  • 32 GB RAM;
  • 6 CPU @ 1.9 Hz
  • Four 2TB SATA disks for the storage pool are combined into two RAID1 arrays;
  • Two SSD drives for the cache pool are combined into a RAID1 array.

Cost of a 6TB Ceph-based storage without buying a data storage system

ComponentsQuantityPrice, USDTotal cost, USD
Supermicro SYS-2029TP-HTR (Dual Socket P (LGA 3647) support, 16 DIMMs; up to 4TB 3DS ECC DDR4-2933MHz† RDIMM/LRDIMM, 6 Hot-swap 2.5" SATA3 drive bays, Up to 2200W Redundant Power Supplies)151405140
RAM 16 GB Kingston DDR4-2666 Single Rank x4 CL19 1.2v ECC Registered DIMM6129774
Xeon Bronze 3204 1.9/1.9 GHz, 6C/ 6T, 8.25 MB L3, DDR4-2133/1.0 TB, 9.6 GT/s UPI(2), 85 W3270810
HDD SATA 2TB 2.5" Enterprise Capacity SATA 6Gb/s 7200rpm 128Mb 512E123153780
SSD 2.5'' 400GB Server Series.61771062
Total:11566

Without using Ceph, you would have had to buy a storage system with disks. The minimum cost would have been about 19,000 USD - 39% higher.

Ceph-based storage deployed on used servers

You can reduce costs even more with used servers. Supermicro servers with 48 GB RAM configurations and 5600 CPU series will cost about 800 USD.

I strongly recommend that you buy new discs. Each server can have one SSD disk for caching and three SATA disks.

Cost of a 6TB Ceph-based storage without buying a data storage system. Based on used servers

ComponentsQuantityPrice, USDTotal cost, USD
SUPERMICRO 2*Xeon 5645, 48 GB RAM (б/у)48003200
HDD SATA 2 Tб, Western Digital RAID EDITION1270840
SSD 2.5' 400 GB DC S3710 Series43151260
Total:5300

Private cloud

Cloud without the use of Ceph
A classic private cloud would require two components:

  • Compute — servers that are used by virtual machines for computing.
  • Storage — a data storage system.

Even a small cloud built according to this scheme will not be cheap. For example, a cloud of three data storage components would cost about 31,000 USD:

19,000 USD - the cost of the data storage system

12,000 USD - the cost of the computing part. That is three servers with a 50% fill rate. Price of each: 4,000 USD

Cost of a private cloud without the use of Ceph

ConfigurationIOPs* (read 70%/write 30%, Block size 4K)Useful capacity, GBPrice per 1 GB, USDTotal cost, USD
Fujitsu ETERNUS DX100 + 12 Nearline SAS 1Tb SFF (RAID10) + Supermicro Twin7606005,231 000

Ceph-based cloud

For a Ceph-based cloud, you can use the disk resources of the computing part as storage. We will use the same Supermicro platform. It will be used for the operation of virtual machines, and for storing information. Accordingly, we will choose a more powerful version:

  • 8 CPU @ 3.2 GHz;
  • 96 GB RAM.

The cloud will still work if one node fails. About 10 virtual machines can be deployed on the platform with configurations of 12 GB RAM, 4 vCPU, 400 GB of disk space.

Cost of a Ceph-based private cloud

ComponentsQuantityPrice, USDTotal cost, USD
Supermicro SYS-2029TP-HTR (Dual Socket P (LGA 3647) support, 16 DIMMs; up to 4TB 3DS ECC DDR4-2933MHz† RDIMM/LRDIMM, 6 Hot-swap 2.5" SATA3 drive bays, Up to 2200W Redundant Power Supplies)151405140
МRAM 16 GB Kingston DDR4-2666 Single Rank x4 CL19 1.2v ECC Registered DIMM181292322
CPU Xeon Silver 4215R 3.2/4.0 GHz, 8C/16T, 11.00 MB L3, DDR4-2400/1.0 TB, 9.6 GT/s UPI(2), 130 W310063018
HDD SATA 2TB 2.5" Enterprise Capacity SATA 6 GB/sec 7200rpm 128 MB 512E123153780
SSD 2.5 400 GB Server Series61771062
Total:15322

Final comparison of the cost of storage with and without Ceph

6 TB storage6 TB private cloud
Storage without CephStorage with CephStorage with Ceph and used serversCloud without CephCloud with Ceph
Total cost, USD19,00011,5665,30031,00015,322

Create a fault-tolerant virtual environment with VMmanager platform and Ceph storage

VMmanager — is a virtualization platform based on QEMU/KVM and LXD/LXC containers.

VMmanager has ready integration with Ceph - disks of virtual machines are stored in a fault-tolerant cluster. In the event of a fault, data will be recovered automatically.

ЗRequest a VMmanager demo to learn more about its features.