Vitastor

• vitastor.io
• Distributed SDS
• Original design goals:
  🚀 Low latency (~0.1 ms)
  🚀 Low CPU usage
• Block + FS and S3
• «Smart», not Network-RAID

3 years of development

• Stability, CI
• Site and documentation
• A lot of new features
• Convenience

What didn't change?

Block storage layer based on the American blueprints 🤣

Old storage layer at a glance

Store contract

• ID: inode:offset, 128 KB objects
• Partial read/write
• Version number++ on every write...
• Atomic write!
  (network is less stable than disk)
• 2-phase write for EC (write → commit)
  (to fight "raid write hole")

Advantages

• No file system overhead
  No Copy on Write overhead
  ⇒ Low CPU usage
• Small meta — 36 B per object (~290 MB / 1 TB)
• Low latency ≈ disk latency
• Journal ↔ write buffer (SSD+HDD)

Drawbacks

• Write Amplification for 4 KB — 3-5x
  
• 2x metadata in memory
  
• EC: 1 uncommitted entry → OSD hangs
• Non HDD friendly (random write)
• (!!!) No space for extensions

Metadata extensions

• Tombstones 🪦
  • Atomic delete
  • Discard
  • Distributed SSD cache (tiering)
• Dynamic data blocks
• Compression
• Local SSD cache (bcache)

The First Idea

The Second Idea

Append to the end

The Second Idea

😊 Amortized writes ⇒ 1x RAM, WA ↓↓

The Second Idea

😢 Need Compaction (LSM?..) ⇒ WA ↑↑, CPU ↑↑

Idea № 3

Instead of Compaction...

We can just track garbage

Side effect

How to avoid?

Put each object entries to a single block...

...But that's not LSMeta anymore, and WA ↑↑

However, the idea attracted me

NonLSM: heap-meta branch

• 65 files changed, 11489 insertions(+), 6655 deletions(-)
• Passes all tests
• cpp-btree → 🤣 swissmap robin_hood_map
• Atomics (on the next slides)
• Result: 176000 randwrite (~1.75x)
• 😢 Ugly hacks (object is limited by 1 block)
• 😢 WA — exactly 3x or 2x

Return to LSMeta

Decided to rewrite it once more

WA 1 is worth the invalidation 😇

Buffer area instead of the journal

Hashmap and linked lists in RAM

Side note

It's absolutely pointless if you don't optimize CPU usage!

Memory copying, (de)serialization, threads, mutexes... 👎👎👎

Write Amplification

Let's make a CoW FS?

Linux 6.11 To Introduce Block Atomic Writes –
Including NVMe & SCSI Support

• New RWF_ATOMIC flag (to avoid request fragmentation)
• New parameters in /sys/block/*/queue/

Atomic writes in NVMe

Samsung PM9A3 😢

Micron 7450 / 7500

Kioxia (Toshiba) CD7-R / CD8-R

More atomicity

• SCSI: WRITE ATOMIC (enterprise)
• NVMe: atomic_block = (AWUPF+1) * block
• SSD: 4 KB is the translation block (sometimes more)

⇒ 4 KB writes on SSD/NVMe are always atomic!
...cool, Vitastor already relies on it 🤣

Overgrown SSDs

122 TB D5-5536 — 16 KB block

HDD

Experiments with Micron 7450

Write xxx KB blocks → pull the cable → check

• USB-NVMe — does not work (buffering?)
• PCIe — OK! but atomic_write_max_bytes = 128 KB — ?
• max_hw_sectors_kb = 128 — ?
• Linux kernel hardcode. iommu on → 128 KB, off → 256 KB
• Conclusion: IT WORKS!

How to utilize it

MySQL: innodb_doublewrite=OFF

Vitastor: ???

How to utilize it

Write Intents

1. "I'm going to update the block, CRC32=..."
2. Write directly to the data disk
3. Check CRC32 if we crash
   Mismatch ⇒ old version
   Matches ⇒ new version

LSMeta + Write Intent branch

• 36 files changed, 3229 insertions(+), 3708 deletions(-)
• WA ~1.0x!
• Micron/Kioxia or and only 4 KB writes
• 2x latency same latency in practice — ~29 μs!

randwrite 4 kb results (fio_blockstore)

Memory usage

Memory usage per 1 TB*

* without checksums (128 MB .. 1 GB per 1 GB more RAM)

Smart SDS with RAI(N) speed

• The release is coming 😊
• WA 1 — not a bottleneck anymore!
• New store is an option
• Need Micron/Kioxia
• Come to the chat for a test build 😇
• Wait a bit before production 😇