Providers

Summary

Providers are the last step in a capsule’s journey to and from physical or cloud storage. They are small adapters that understand precisely how to store and retrieve capsules in a particular storage system or service, such as local disk or cloud storage. Providers do not know what’s inside capsules. In fact, when providers write and read capsules, capsules are already fully encrypted and opaque containers in a universal format that is totally insensitive to the provider’s storage format.

You are free to use any provider, in any capsule hub, in any combination within a continuum. Except for special providers such as AWS DynamoDB, which have capsule size restrictions, the use of one provider versus another is a question of personal choice and objectives.

Prefixes and Namespaces

Providers are bound to an actual storage system, such as a local filesystem or cloud storage bucket. You can further subdivide a provider’s primary storage with these two parameters:

prefix

The prefix is a name that is added to the path to all capsules stored and retrieved by a provider. You can use a prefix to subdivide, for example, a bucket between different users or continuums without reconfiguring cloud storage. A provider never accesses capsules or other pieces of data (for example, an object) outside of this prefix. If desired, you can configure security profiles in the cloud service to further restrict objects, files or database records to only those that start with the configured prefix.

namespace

A namespace is an additional subdivision of a provider and prefix. It allows a provider and prefix to be further shared between users and continuums. Namespaces are supplemental divisions used primarily by gateways (see below) to deploy private cloud infrastructure.

Using Local Storage

When using local storage for capsule storage, you configure a provider of type ‘file’, and give the provider the path to a filesystem location where capsules will be stored. You can also use a ‘prefix’ for local storage, but this simply becomes an additional level added to the path of a capsule - you can use either method to configure where capsules are stored in local storage.

Structure

Capsules are stored in local storage as files under encoded directory structures. All directory names, file names and other contents are fully encrypted inside capsules and, therefore, do not disclose any information about the data they contain. You can implement whatever local security profile you find appropriate to protect these files; all capsules are written under the UID that is running the cosnim command that is accessing the continuum.

Making Local Storage Immutable

You can make local storage immutable by using gateways. Gateways are small Cosnim network components that serve as firewalls between Cosnim instances and physical storage. They can prevent Cosnim instances and other processes from damaging or destroying capsules in storage providers as an additional shield against cyber and ransomware attacks.

Clean-up and Purges

Because local storage often has limited capacity, you may want to monitor local storage use and may wish to destroy capsules stored in local storage as follows:

Staging Hubs Cleanup

In staging hubs, you can use the ‘safe_to_purge’ parameter (see Configuration Reference) to have Cosnim generates a list of capsule files that you may delete from the staging hub’s local storage cache. You can delete any capsule listed without further coordination with Cosnim.

Scatter Hub Cleanup

In scatter hubs, you can purge local capsules to free up space by draining the hub with the command:

cosnim drain hub --hub NAME ...

This will cause Cosnim to migrate capsules to other hubs according to the scatter hub’s configuration.

Cloud Object Storage

Cloud object storage is the most common and flexible form of cloud storage. It is used simply by configuring a provider to use a particular cloud service. Although the parameters of a cloud provider may vary from one service to another, the process generally remains the same:

Configure the cloud storage service, for example, a cloud object storage bucket, where the provider will store capsules.
Generate access credentials (such as an access and secret key) appropriate to the cloud provider. The credentials should give Cosnim the right to write, list and read objects, records, or whatever format is appropriate for that cloud storage service.
Save the credentials in the keychain with the command:
```
cosnim import key --key KEYNAME --file FILENAME
```
Refer to the Configuration Reference for the credentials format to use with each cloud provider.
Configure the provider under the storage hub’s configuration, referring to the bucket and credentials previously created.

Once configured, you need to initialize the cloud storage with either command:

cosnim create continuum       # New continuum
cosnim expand continuum       # Add new hubs to an existing continuum

Capsules in cloud storage are no different than capsules in local storage. You can export and import capsules from cloud storage using any available tool. As long as the capsules belong to the same continuum and hub level (for example, a scatter hub), you can freely copy, move, back up and restore capsules directly to and from cloud storage without notifying Cosnim.

You should consider using the cosnim drain command to ask Cosnim to move capsules out of a storage hub to another one to redistribute storage use. Using the drain command helps to avoid cloud storage access retries in other running Cosnim instances using the same continuum.

Other Cloud Storage

There are other non-object cloud storage providers you can use. These include, for example, Dropbox and AWS DynamoDB. You can use these alternate providers for special purposes:

Dropbox is useful for leveraging inexpensive storage. This particular provider is not ideal for multi-user continuums due to its limited API, but can be a valuable tool to reduce storage costs.

AWS DynamoDB can quickly store and retrieve small capsules, such as control information, which is useful when sharing a continuum with multiple users. In addition, DynamoDB supports atomic operations, something that is not available in other cloud providers and which is necessary to control the sharing of a filesystem at share level 1 and above. You don’t need to use DynamoDB; you may also use Cosnim Gateways in front of existing storage to get the same results.