Fast and simple distributed
software-defined storage

Vitastor

  • Distributed SDS
  • ...but Fast 🚀 — latency 0.1 ms
  • ...and Simple ✌️ — 60k lines of code, not 1 mln
  • From Russia with love 🙂

Software-Defined Storage (SDS)

Software that assembles usual servers with usual drives into a single scalable fault-tolerant storage cluster with extended features

Why do IaaS/PaaS providers use SDS?

  • Scalability
  • Client data preservation
  • Reduced costs due to hyperconvergence
  • No need for server "micromanagement"
  • Different storage classes (SSD, HDD)

But most SDS are an overhead

Overhead

  • Q=1 — best possible latency
  • 4 kb write to SSD — 0.04 ms
  • 4 kb write to Ceph — from ~ 1 ms
  • Internal cloud SDS's – ± the same
  • 2400 % overhead !

Vitastor

  • ~ 0.1 ms replicated latency
  • ~ 0.2 ms erasure-coded latency
  • 3-8 GByte/s per client (VM)
  • ~ 400000 iops per client
  • Just ~1 CPU core per NVMe disk
  • Low (50%) rebalance impact

Vitastor — protocols

  • Block access (VM disks, containers):
    Kubernetes, Proxmox, OpenNebula, OpenStack and others
  • VitastorFS (NFS) — clustered POSIX FS
  • Object storage (S3) — based on Zenko CloudServer

Features

  • Replication, erasure codes (N+K)
  • All disk support: SSD/NVMe, HDD, SSD+HDD
  • Flexible data placement
  • RDMA / RoCEv2 support
  • Fast snapshots and clones
  • Fast checksums, data scrubbing
  • Monitoring
  • Kubernetes operator
  • And more...

Architecture

  • Symmetric distributed, Ø SPOF
  • Block base layer
  • Uniform load balancing
  • Transactional writes → data loss protection
  • Optimised for modern SSD/HDD

Ease of support

  • Low number of components
  • Human-readable metadata in etcd
  • Minimal external dependencies
  • Compact implementation:
    ~60k lines of code (Ceph ~1M)
  • Non-standard architecture support (E2K)

Licensing

  • Own Copyleft license: VNPL
  • Free use in OpenSource environment
  • Closed-source services require commercial support
  • Technical and architectural support from author

Contacts

< 0.1 ms

Random Q1 read/write with fsync

400k iops per client

Random Q128 read/write
(180k in QEMU, bounded by virtio)

Up to 8 GB/s per VM

Linear read/write per client (VM)

~1 CPU per NVMe

0.3-2 CPU, varies with I/O mode and TCP/RDMA network

Block Storage

With support for all major KVM-based systems and containers: OpenNebula, OpenStack, Proxmox VE, Kubernetes

Clustered File System

Ground-up implementation of a scalable POSIX
Read-Write-Many file system, mountable over NFS 3.0

Object Storage (S3)

Based on Zenko CloudServer

Latest Posts

Vitastor 3.0.13 released

  • New store fixes:
    • Fix repeated rollback logic
    • Fix crash on rolled back object compaction
    • Fix postpone_load possibly merging different object chains
    • Fix block_csums import in vitastor-disk write-meta
    • Fix header checksum after vitastor-disk write-meta
  • Old store fixes:
    • Fix batched fsync possibly skipped by some flush coroutines
  • Improve ENOSPC test, fix possible crash on ENOSPC
  • Add fsyncs to vitastor-disk prepare
  • Fix possible crash on pg_lock check failure in sec_read_bmp
  • Fix VitastorFS initialization when local_reads are enabled

2026-05-31 Continue reading →

Vitastor 3.0.12 released

Important fixes (except the new store)

  • Fixed a possible use-after-free in the OSD during error handling of initial commit/rollback of objects in EC pools.
  • Fixed a possible free of an invalid pointer in the OSD during read errors from snapshot/clone chains in EC pools.
  • Fixed possibly incorrect handling of commit/rollback operations in EC pools during pool PG count changes.
  • Fixed the inverted fsync enable parameter in the ublk driver (fsync was not enabled on pools without immediate_commit).
  • Added the raw-ls command for debugging purposes to find object versions in the cluster using listing operations.

2026-05-17 Continue reading →

Vitastor 3.0.11 released

A single hotfix:

  • Fix missing bitmap read error handling in EC reads from cloned images or from images with snapshots potentially leading to invalid (zero) reads during OSD restarts / PG repeerings

⚠️ Warning - if you’re affected, ideally you should pause client I/O during update

2026-05-09 Continue reading →

Vitastor 3.0.10 released

Important bug fixes (new store):

  • Fix OSDs possibly refusing to start with “write metadata failed at offset xxx: Invalid argument” (fix buffer alignment during initial garbage collection)
  • Rollback change from 3.0.4 - on-disk garbage entries are not skipped on start again. This change doesn’t have any impact normally, but OSDs originally running 3.0.0-3.0.2 and then upgraded to 3.0.9 may hit a bug where 3.0.9 refuses to start due to entries marked as garbage too early and flushed to disk in 3.0.0-3.0.2.

Other changes:

  • Auto-select the only RDMA device/port if there is only one
  • Rollback one 3.0.9 change - there was no actual use-after-free :) (the problem was only relevant to an unstable development version)
  • Fix vitastor-nfs --trace option
  • Fix an unintended 1 second sleep in vitastor-cli rm-data
  • Fix inode statistics not being cleared for a deleted pool
  • Fix print to stdout in client

2026-04-27 Continue reading →

Vitastor 3.0.9 released

  • Fix clients reading from snapshots incorrectly on retries (for example when OSDs were restarted during read)
  • Fix a possible “send-after-free” in OSD. It turned out that this problem was never present in any released versions and the fix was only relevant for an unstable development branch. The change will be rolled back in 3.0.10.
  • Fix broken garbage collection in the new store leading to very high memory usage
  • Make the new store do full garbage collection on OSD startup by default (can be turned off with a new gc_on_start parameter)
  • Make startup logging with the new store more informative, allow to log amount of garbage with log_level > 1
  • Fix OSD crashing when handling a read_bitmap operation from a deleted object with the new store

2026-04-04 Continue reading →

All Posts