S3 implementation comparison

One question: Where to pilfer which S3 implementation to reuse?

I have no desire to write S3 from scratch, because the protocol, even though simple, has a lot of small details that require attention and cannot be ignored, because users always want maximum compatibility with Amazon S3.

At the time of writing this post, the following options were found: Minio, SeaweedFS, Ceph (RadosGW), Zenko CloudServer, OpenStack Swift, and Deuxfleurs Garage.

The main interest during comparison is comparing the S3 frontend, the external part of the server, because the storage layer will anyway be replaced with our own one (Vitastor).

S3 tests from Ceph were run during comparison, but in the simplest configuration, without setting up additional integrations possibly required for some of the tests.

Let’s try to look at the features of each implementation!

2024-05-09 Continue reading →

Vitastor 1.6.1 released

A bunch of monitor fixes

• Add noout flag for OSDs (/vitastor/config/osd/xx)
• Fix “effective” size of degraded PGs (and thus “used space”) calculation in monitor
• Fix monitor not clearing PGs of deleted pools
• Fix incorrect PG generation with hosts with 0 OSDs
• Fix monitor crashing during primary OSD recheck when pool has no PGs
• Fix monitor crashing when node_placement included non-existing OSDs
• Fix possible data movement after removing OSDs reweighted to 0
• Remove extra empty keys from pool configurations created by vitastor-cli create-pool
• Fix 32-bit build

2024-04-22 Continue reading →

Vitastor 1.6.0 released

Hierarchical failure domains, handling of 100% full OSDs, nbd-netlink and more.

2024-04-09 Continue reading →

Vitastor 1.5.0 released

After half a year of hard work, VitastorFS is finally here ! :-)

2024-03-16 Continue reading →

Vitastor 1.4.8 released

• Do not use \r if output is not a terminal (should fix unexpected job output in proxmox)
• Fix rm/rm-data error return code, add --down-ok option to bypass the error
• Add EIO retry timeout and allow to disable these retries, rename up_wait_retry_interval to client_retry_interval
• Add ubuntu jammy build
• Wait for blockstore initialisation before starting OSD (prevent timeouts when init takes time)
• Fix a rare use-after-free in automatic sync after delete in blockstore

2024-02-29 Continue reading →

Vitastor 1.4.7 released

• Fix another old “BUG: Attempt to overwrite used offset” in a very simple case: bs=4k rw=write iodepth=16 from OSD start; add this case to tests
• Fix a rare crash with “unexpected state during flush: 0x51” possible with EC since 1.4.2 during rebalance and OSD outages
• Fix a rare write stall with EC & immediate_commit=none caused by sync operations reserving unneeded space in the journal
• Fix 32-bit build warnings, most in printf/scanf format strings

2024-02-22 Continue reading →

Vitastor 1.4.6 released

Unwavering stabilization of 1.4.x, continued :-)

• Include the accidentally lost part of 1.4.5 journal trimming fix
• Fix a possible OSD crash with “BUG: Attempt to overwrite used offset” which was probably present for long time, but became apparent after fixing flapping tests in CI
• Fix remaining flapping tests in CI. It was the first time when tests actually passed without retries :-)

2024-02-20 Continue reading →

Vitastor 1.4.5 released

• Fix a write stall caused by incorrect journal trimming introduced in 1.4.4 :)
• Fix PGs sometimes hanging in “starting” state on mass OSD restarts
• Fix a rare crash with “map::at” during OSD pings
• Use new defaults for non-capacitor (desktop) SSDs - improves T1Q256 random write from ~6k iops to ~45k iops
• Make journal_trim_interval configurable

2024-02-16 Continue reading →

Vitastor 1.4.4 released

A couple of fixes for EC pools

• Fix a segfault possible on partial EC overwrite in 1234 -> 5030 rebalance scenario
• Fix two problems leading to EC pools stalling on rebalance & parallel sudden stops of OSDs, for example during a sudden poweroff of a host:
  • Recovery auto-tuning (1.4.0 feature) could apply too large delays and stall the EC journal - fixed by limiting delays with a new recovery_tune_sleep_cutoff_us parameter (10 seconds by default) and applying recovery pauses before write operations, not after them, to not occupy space in the journal for long time
  • Dynamic journal space reservation (1.3.0 feature) wasn’t accounting new writes when checking the limit so OSDs could still fill the journal fully and stall - fixed by including new writes into the limit
• Print etcd dbSize instead of dbSizeInUse in status

2024-02-11 Continue reading →

Vitastor 1.4.3 released

Hotfix for hotfix O:-)

• “Write stall fix” was incomplete and EC write stalls could continue even on 1.4.2. Now they’re finally fixed O:-)
• Make monitor ignore statistics of stopped OSDs. Previously if you stopped all OSDs the last total I/O numbers would remain the same indefinitely

2024-02-09 Continue reading →