Skip to content

A distributed key-value storage system developed and maintained by Xiaomi Cloud Storage Team.

License

Notifications You must be signed in to change notification settings

empiredan/pegasus

This branch is 6 commits ahead of, 21 commits behind apache/incubator-pegasus:master.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

e090da4 · Dec 3, 2024
Jun 29, 2023
Nov 25, 2024
Sep 19, 2024
Oct 21, 2024
Nov 15, 2024
Sep 25, 2024
Sep 19, 2024
Nov 28, 2024
Jul 26, 2022
Sep 24, 2024
Dec 3, 2024
Nov 28, 2024
Jul 12, 2024
Sep 19, 2024
Sep 23, 2024
Jun 21, 2024
Jan 5, 2024
Dec 3, 2024
Sep 25, 2024
Feb 20, 2024
Jul 11, 2024
Nov 15, 2024
Jun 20, 2024
Apr 24, 2022
Sep 25, 2024
Dec 5, 2023
May 30, 2024
Dec 5, 2023
Dec 1, 2023
Aug 14, 2024
Nov 2, 2023
Dec 4, 2022
Sep 25, 2024

Repository files navigation

pegasus-logo

Lint and build regularly License Releases

Note: The master branch may be in an unstable or even in a broken state during development. Please use GitHub Releases instead of the master branch in order to get stable binaries.

Apache Pegasus is a distributed key-value storage system which is designed to be:

  • horizontally scalable: distributed using hash-based partitioning
  • strongly consistent: ensured by PacificA consensus protocol
  • high-performance: using RocksDB as underlying storage engine
  • simple: well-defined, easy-to-use APIs

Background

Pegasus targets to fill the gap between Redis and HBase. As the former is in-memory, low latency, but does not provide a strong-consistency guarantee. And unlike the latter, Pegasus is entirely written in C++ and its write-path relies merely on the local filesystem.

Apart from the performance requirements, we also need a storage system to ensure multiple-level data safety and support fast data migration between data centers, automatic load balancing, and online partition split.

Features

  • Persistence of data: Each write is replicated three-way to different ReplicaServers before responding to the client. Using PacificA protocol, Pegasus has the ability for strong consistent replication and membership changes.

  • Automatic load balancing over ReplicaServers: Load balancing is a builtin function of MetaServer, which manages the distribution of replicas. When the cluster is in an inbalance state, the administrator can invoke a simple rebalance command that automatically schedules the replica migration.

  • Cold Backup: Pegasus supports an extensible backup and restore mechanism to ensure data safety. The location of snapshot could be a distributed filesystem like HDFS or local filesystem. The snapshot storing in the filesystem can be further used for analysis based on pegasus-spark.

  • Eventually-consistent intra-datacenter replication: This is a feature we called duplication. It allows a change made in the local cluster accesible after a short time period by the remote cluster. It help achieving higher availability of your service and gaining better performance by accessing only local cluster.

To start using Pegasus

See our documentation on the Pegasus Website.

Client drivers

Pegasus has support for several languages:

Contact us

  • Send emails to the Apache Pegasus developer mailing list: dev@pegasus.apache.org. This is the place where topics around development, community, and problems are officially discussed. Please remember to subscribe to the mailing list via dev-subscribe@pegasus.apache.org.

  • GitHub Issues: submit an issue when you have any idea to improve Pegasus, and when you encountered some bugs or problems.

Related Projects

Test tools:

Data import/export tools:

License

Copyright 2022 The Apache Software Foundation. Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

About

A distributed key-value storage system developed and maintained by Xiaomi Cloud Storage Team.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • C++ 75.4%
  • Java 8.8%
  • Go 6.2%
  • Shell 2.7%
  • Python 1.8%
  • CMake 1.5%
  • Other 3.6%