VoluSnap — Cloud Volume Auto Snapshot Go Server
VoluSnap — Cloud Volume Auto Snapshot Go Server
Do you deploy your web apps to cloud virtual machines such as Digital Ocean Droplets or Scaleway Cloud Instances ?
If your app saves any kind of data to disk and you’re worried about data loss due to software bugs, human errors or malicious actors (amongst other risks) then you need a backup strategy.
Cloud Volume Backups
Most cloud providers provide a Hot Backup feature which takes a snapshot of your volume data without the need to stop your instances during the backup process (but there is a data loss risk involved, please read up on this topic and make an informed decision).
However, automated recurring backups are not always available, and sometimes the configuration options are not granular enough. At the time of writing this article Digital Ocean Backups are taken once a week, and Scaleway only offers manual backups.
Fortunately, most cloud providers provide API access. I’ve needed an automated backup solution for my personal cloud deployments for a while, so I decided to build an open source solution:
VoluSnap: A self hosted solution built with Go
Volusnap is an API server built with Go. It allows triggering automated recurring snapshots of cloud provider volumes. Here is a simplified schema explaining how VoluSnap works:
The basic workflow is :
User signs up
User logs in, generating a JWT token
User adds a Cloud Account and relevant Cloud API access token
User lists Cloud Volumes
User creates Snapshot Rule(s) for the chosen Volume(s)
Rules Checker Service checks the snapshot rules periodically, triggers snapshots via Cloud Provider API and saves snapshot Metadata to the Database
VoluSnap uses:
gorilla/mux for routing
sqlboiler for data persistence to PostgreSQL
sql-migrate for database migrations
viper for configuration
testify for unit tests and mocks
Components:
REST API
SnapRules Checker Service
The REST API is documented in the github repo. No client code is provided at the moment.
At the moment both the REST API and the SnapRules Checker components run in the same Go process. The SnapRules Checker Service runs in a goroutine at the moment but it would be doable to split it into its own process in the future if needed:
func (checker *snapRulesChecker) Start() {
logrus.Infof("Starting snapRulesChecker ...")
checker.ticker = time.NewTicker(5 * time.Minute)
go func() {
for {
select {
case <-checker.ticker.C:
logrus.Infof("Checking SnapRules ...")
err := checker.checkAll()
if err != nil {
logrus.Errorf("checkall snaprules err: %v", err)
}
case <-checker.stop:
return
}
}
}()
}
func (checker *snapRulesChecker) Stop() {
logrus.Infof("Stopping snapRulesChecker ...")
checker.ticker.Stop()
close(checker.stop)
}Adding Cloud Providers
VoluSnap currently supports Digital Ocean and Scaleway. I’ve tried to make adding other Cloud providers easy (Hint: PRs welcome ❤️). As you can see in the code below, pRegistry is a package level singleton service you can register provider services with. Your provider just has to implement the interface ProviderSvcer :
type ProviderSvcer interface {
ListVolumes() ([]Volume, error)
TakeSnapshot(snapRule *models.SnapRule) (string, error)
}package api
import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"strconv"
"time"
"github.com/didil/volusnap/pkg/models"
)
func init() {
// self register with provider registry
pRegistry.register("digital_ocean", newDigitalOceanServiceFactory())
}
func newDigitalOceanServiceFactory() *digitalOceanServiceFactory {
return &digitalOceanServiceFactory{}
}
type digitalOceanServiceFactory struct{}
func (factory *digitalOceanServiceFactory) Build(token string) ProviderSvcer {
return &digitalOceanService{token: token, rootURL: "https://api.digitalocean.com/v2"}
}
type digitalOceanService struct {
token string
rootURL string
}
func (svc *digitalOceanService) ListVolumes() ([]Volume, error) {
req, err := http.NewRequest(http.MethodGet, svc.rootURL+"/droplets", nil)
if err != nil {
return nil, fmt.Errorf("DO list droplets NewRequest err: %v", err)
}
req.Header.Set("Authorization", "Bearer "+svc.token)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("User-Agent", "VoluSnap")
client := &http.Client{Timeout: 30 * time.Second}
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("DO list droplets req err: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
body, _ := ioutil.ReadAll(resp.Body)
return nil, fmt.Errorf("DO list droplets %v : %v", resp.Status, string(body))
}
type dropletRegion struct {
Slug string `json:"slug,omitempty"`
}
type droplet struct {
ID float64 `json:"id,omitempty"`
Name string `json:"name,omitempty"`
Disk float64 `json:"disk,omitempty"`
Region dropletRegion `json:"region,omitempty"`
}
type dropletsList struct {
Droplets []droplet `json:"droplets,omitempty"`
}
var b dropletsList
err = json.NewDecoder(resp.Body).Decode(&b)
if err != nil {
body, _ := ioutil.ReadAll(resp.Body)
return nil, fmt.Errorf("DO list droplets json decode err: %v , body: %v", err, body)
}
var volumes []Volume
droplets := b.Droplets
for _, d := range droplets {
volumes = append(volumes, Volume{
ID: strconv.Itoa(int(d.ID)),
Name: d.Name,
Size: d.Disk,
Region: d.Region.Slug,
})
}
return volumes, nil
}
type doTakeSnapshotReq struct {
Type string `json:"type,omitempty"`
}
func (svc *digitalOceanService) TakeSnapshot(snapRule *models.SnapRule) (string, error) {
var r bytes.Buffer
json.NewEncoder(&r).Encode(&doTakeSnapshotReq{Type: "snapshot"})
req, err := http.NewRequest(http.MethodPost, svc.rootURL+"/droplets/"+snapRule.VolumeID+"/actions", &r)
if err != nil {
return "", fmt.Errorf("DO TakeSnapshot NewRequest err: %v", err)
}
req.Header.Set("Authorization", "Bearer "+svc.token)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("User-Agent", "VoluSnap")
client := &http.Client{Timeout: 30 * time.Second}
resp, err := client.Do(req)
if err != nil {
return "", fmt.Errorf("DO TakeSnapshot req err: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
body, _ := ioutil.ReadAll(resp.Body)
return "", fmt.Errorf("DO TakeSnapshot %v : %v", resp.Status, string(body))
}
type action struct {
ID float64 `json:"id,omitempty"`
Status string `json:"status,omitempty"`
}
type actionResp struct {
Action action `json:"action,omitempty"`
}
var a actionResp
err = json.NewDecoder(resp.Body).Decode(&a)
if err != nil {
body, _ := ioutil.ReadAll(resp.Body)
return "", fmt.Errorf("DO TakeSnapshot json decode err: %v , body: %v", err, body)
}
providerSnapshotID := strconv.Itoa(int(a.Action.ID))
return providerSnapshotID, nil
}And don’t forget the corresponding unit tests
package api
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"github.com/didil/volusnap/pkg/models"
"github.com/stretchr/testify/assert"
)
func Test_digitalOceanService_ListVolumes(t *testing.T) {
token := "my-token"
factory := newDigitalOceanServiceFactory()
doSvc := factory.Build(token).(*digitalOceanService)
volumes := []Volume{
Volume{ID: "3164444", Name: "example.com", Size: 25, Region: "nyc3"},
Volume{ID: "95874511", Name: "my-other-droplet", Size: 50, Region: "nyc1"},
}
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/droplets")
assert.Equal(t, "Bearer "+token, r.Header.Get("Authorization"))
w.Header().Set("Content-Type", "application/json")
w.Write([]byte(`{
"droplets": [
{
"id": 3164444,
"name": "example.com",
"memory": 1024,
"vcpus": 1,
"disk": 25,
"locked": false,
"status": "active",
"volume_ids": [ ],
"size": { },
"size_slug": "s-1vcpu-1gb",
"region": {
"name": "New York 3",
"slug": "nyc3",
"sizes": [
],
"features": [
"virtio",
"private_networking",
"backups",
"ipv6",
"metadata"
],
"available": null
}
},
{
"id": 95874511,
"name": "my-other-droplet",
"memory": 2048,
"vcpus": 1,
"disk": 50,
"locked": false,
"status": "active",
"volume_ids": [ ],
"size": { },
"size_slug": "s-1vcpu-1gb",
"region": {
"name": "New York 1",
"slug": "nyc1",
"sizes": [
],
"features": [
"virtio",
"private_networking",
"backups",
"ipv6",
"metadata"
],
"available": null
}
}
]
}`))
}))
defer s.Close()
doSvc.rootURL = s.URL
myVolumes, err := doSvc.ListVolumes()
assert.NoError(t, err)
assert.ElementsMatch(t, myVolumes, volumes)
}
func Test_digitalOceanService_TakeSnapshot(t *testing.T) {
token := "my-token"
factory := newDigitalOceanServiceFactory()
doSvc := factory.Build(token).(*digitalOceanService)
snapRule := &models.SnapRule{VolumeID: "vol-3"}
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/droplets/"+snapRule.VolumeID+"/actions")
assert.Equal(t, "Bearer "+token, r.Header.Get("Authorization"))
var reqJSON doTakeSnapshotReq
err := json.NewDecoder(r.Body).Decode(&reqJSON)
assert.NoError(t, err)
assert.Equal(t, reqJSON.Type, "snapshot")
w.Header().Set("Content-Type", "application/json")
w.Write([]byte(`{
"action": {
"id": 36805022,
"status": "in-progress",
"type": "snapshot",
"started_at": "2014-11-14T16:34:39Z",
"completed_at": null,
"resource_id": 3164450,
"resource_type": "droplet",
"region": {
"name": "New York 3",
"slug": "nyc3",
"sizes": [
"s-1vcpu-3gb",
"m-1vcpu-8gb",
"s-3vcpu-1gb",
"s-1vcpu-2gb",
"s-2vcpu-2gb",
"s-2vcpu-4gb",
"s-4vcpu-8gb",
"s-6vcpu-16gb",
"s-8vcpu-32gb",
"s-12vcpu-48gb",
"s-16vcpu-64gb",
"s-20vcpu-96gb",
"s-1vcpu-1gb",
"c-1vcpu-2gb",
"s-24vcpu-128gb"
],
"features": [
"private_networking",
"backups",
"ipv6",
"metadata",
"server_id",
"install_agent",
"storage",
"image_transfer"
],
"available": true
},
"region_slug": "nyc3"
}
}`))
}))
defer s.Close()
doSvc.rootURL = s.URL
providerSnapshotID, err := doSvc.TakeSnapshot(snapRule)
assert.NoError(t, err)
assert.Equal(t, providerSnapshotID, "36805022")
}Conclusion
I hope that you’ll find VoluSnap useful for your Cloud Volume Backups. Definitely leave comments if you like/hate/have questions about the project or the code/testing patterns used !