Eagle
Eagle is a lightweight and intelligent p2p based docker image distribution system.
Features
- Non-invasive: Eagle can seamlessly support docker for distributing images.
- High-availability: Eagle supports high-availability in both client-side and server-side, no component is a single point of failure.
- SSI(Seeder Storage Interface). Eagle Seeder plugs into reliable blob storage options, like local FileSystem or S3. The seeder storage interface is simple and new options are easy to add.
- Peer optimal arithmetic: Eagle supports peer optimal arithmetic to improve performance and save cross-IDC bandwidth.
- Host level speed limit: Many downloading tools(wget/curl) only have rate limit for the current download task, but Eagle also provides rate limit for the entire host.
- LRUCache delete policy: Both Seeder and EagleClient achieves the LRUCache delete policy.
- Lightweight: Eagle consists of only several necessary components, which makes it understandable, maintainable and easy-to-use.
Architecture
The principle of eagle is kept as simple as possible and can be illustrated as follows:
- Proxy
- Deployed on every host
- Proxy the blob request(EagleClient => Original Request)
- EagleClient
- Announces available content to tracker
- Connects to peers returned by tracker to download or upload content
- Seeder
- Stores blobs as files on disk backed by pluggable storage (e.g. FileSystem, S3)
- Provides meta info of blob to EagleClient and acts as the first uploader
- Tracker
- Tracks which peers have what content (both in-progress and completed)
- Provides ordered lists of peers to connect to for any given blob
- Origin
- Docker Distribution
Comparison With Other Projects
Dragonfly from Alibaba
Dragonfly provides many great features, such as interruption resuming capability, Host level speed limit and so on, which makes it the most popular p2p based image distribution solution. And more recently it becomes the CNCF Incubating Project.
One drawback of Dragonfly is that it doesn’t support high-availability, and its central supernode design makes performance degrade linearly as either blob size or cluster size increases.
kraken from uber
Kraken uses several components, such as Agent, Origin, Tracker, Proxy and Build-Index, combined with its own designed driver protocol to build a p2p distribution system.
Eagle uses almost the same components with kraken, but it is more compact and simple as it uses BitTorrent protocol underlayer and drops some unessential components.
Suggestion
In my opinion, both Dragonfly and Kraken are a good option inside a large, established enterprise since each of them is complicated enough and has its own set of advantages. However I would recommend the Eagle if you want a more flexible and lightweight solution since eagle is absolutely more compact, simple and maintainable compared with them.
TODO
- Peer optimal arithmetic
- Push notification mechanism