External S3 storage

BILLmanager supports integration with external S3-compatible object storage based on Ceph RADOS. This allows you to provide cloud storage services to your clients. Features of the service:

• the service takes into account the volume and number of buckets;
• the service provides API access;
• payment for the service is charged based on actual usage.

Key concepts

S3 (Simple Storage Service) is a standard for storing and accessing data in object format. Instead of a file system with folders and disks, object storage is used, where data is stored in the form of:

• objects — data (files), metadata, and unique IDs;
• buckets — containers in which objects are placed. Analogous to top-level directories. Buckets help group data by project, application, or data type. Each bucket has a unique name within the storage. For example, my-backups, client-asset. When configuring, you can set individual access parameters, policies, and data management rules for it. A bucket can be:
  • public — accessible via a link;
  • private — accessible via keys;

A bucket can be configured as hosting for a static website — for example, to display a landing page, documentation, or SPA.

Access keys — a pair of Access Key ID and Secret Key — are used for authentication and access to private data.

S3 Management in BILLmanager

Connecting the Ceph S3 handler

To create a service handler, go to Integration → Processing modules → Add button:

1. Product type — select External S3 storage . For more information about product types, see Product type parameters.
2. Processing module — select Ceph S3 → Add button.
3. Integration settings. Please fill in the required fields:
   • URL to access to the S3-service API. For example, https://s3.example.com;
   • User — user name or account ID;
   • Access key — storage key identifier;
   • Secret key — storage secret key.
4. If necessary, please specify:
   
   • Region (location) — storage location;
   •  Links — please specify if you plan to connect sites to buckets:
     • Link to bucket (path-style) — for example,   https://s3.example.com/%s ,  where %s is the bucket name;
     • Link to bucket (domain-style) — for example, https://s3.example.com/%s ,  where %s is the bucket name;
     • Link to a bucket (website) — for example,  https://s3site.example.com/%s ,   where %s is the bucket name;
5. Click Next.
6. Configure Processing module parameters:
   1. Data center — select the data center in BILLmanager to which the processing module will be connected.
   2. Name — specify the name of the processing module to be displayed in the BILLmanager interface.
   3. Department — select the department to which tasks from the processing module will be assigned.
   4. Sort — specify the processing module’s priority. If more than one processing module is connected to a tariff plan, the processing module with the lowest priority is selected first when creating the service.

Setting the tariff

To configure a tariff, go to the Products → Tariff plans → Add button:

1. Product type — select External S3 Storage .
2. Tariff plan parameters:
   1. Processing module — select the created processing module for Ceph S3.
   1. Processing module settings:
      
      1. Select the Create all add-ons option.

For more information on creating tariff plans, see the article Creating, customizing, archiving a tariff.

Setting up tariff content

After creating a tariff, configure the tariff content. To do this, go to  Products → Tariff plans → select the created tariff → Options button. For more information on how to configure, see the article How to create and configure a tariff plan article.

Product type parameters

To manage the default settings, go to Products → Product types → External S3 Storage → select the parameter to change → click Edit. For more information, see Product types.

Basic product type parameters:

The product content may include parameters:

Working with buckets

To manage buckets, go to the menu Products/Services → External S3 storage → select the ordered tariff → Buckets button. You can:

• Create a bucket;
• change bucket Options:
  • select Access type:
    • Private — access to the contents is possible only with the use of access keys;
    • Public-read — the contents of the bucket are available via a direct link;
  • Limit size of the bucket— specify the Size limit of the bucket, GiB. Exceeding the limit will block the upload of new files;
  • Limit number of objects — specify the Limit on the number of objects, pcs. When the limit is reached, it will be impossible to add new objects;
• Delete bucket;
• view Access — the section displays:
• Bucket information — bucket ID and links to the bucket;
• Account owner's authorization details — URL, connection port, key ID and secret key for accessing the S3 storage, username and region;
• configure a static Website. You can:
  • enable the static Website option for the bucket. After enabling, fill in the Start page and Error page fields;
  • Redirect all requests to the specified domain. You must specify the domain to which the Redirection will occur and select the HTTP or HTTPS protocol;

For the site to work correctly, allow access to the bucket files that are used for the site and are included in the HTML page. To do this:

1. 1. Connect to the storage using the client. For more information, see the Connecting to storage section of this article.
   2. Select the files you need.
   3. Select the Allow access option.

The static website settings may differ depending on the Ceph version and installation methods. For example, if you are using IBM Storage Ceph v.8.0.0, to enable Ceph Object Gateway for static web hosting, set the parameters described in the documentation.

More actions with the service

To perform additional actions with the service, go to the Products/Services menu → External S3 storage → select the ordered service. You can:

• change access keys — Access button → Change keys button. The updated Access key and Secret key will be available in the Access data for connecting to the server subsection;
• set up renewal — Access button → select the renewal cost or disable it;
• change the service tariff — Change tariff button → select New tariff plan. To select from tariffs of all providers, select the Show all tariff plans → Show tariff plans associated with other providers option;
• delete the tariff — the storage will become unavailable to the client;
• activate or deactivate the tariff using the Enable and Disable buttons.

Connecting to storage

Use an S3 client to manage files and buckets. Below are instructions for different OS.

macOS and Windows (using Cyberduck as an example)

Cyberduck is a graphical client for working with S3 storage. Management is carried out through an interface similar to a file manager.

1. Download the client from the official website: https://cyberduck.io/download/
2. Install the application.
3. Launch Cyberduck.
4. In the Open Connection menu, select the Amazon S3 connection type.
5. Fill in the fields:
   • Server / Endpoint: URL of the S3-compatible storage;
   • Port: 443;
   • Access Key ID: Storage key identifier;
   • Secret Access Key: Storage secret key;
   • Disable the Anonymous Login option.
6. Click Connect.

Linux: AWS CLI

To work with S3 storage on Linux, use the command line (AWS CLI):

1. Download and install AWS CLI v2:

   Installation commands

   curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
   unzip awscliv2.zip
   sudo ./aws/install

2. Configure access:

   1. Enter the command:

      aws configure

   2.  In the interactive shell, specify:

      • Access Key: Storage key identifier;
      • Secret Key: Storage secret key;
      • Default region name [None]: press Enter to skip;
      • Default output format [None]: press Enter to skip.

Commands for working with storage:

aws s3 ls --endpoint-url https://<handler_URL> --no-verify-ssl

Where:

• <handler_URL> — the URL of the S3-compatible storage.

aws s3 cp ./localfile.txt s3://mybucket/ --endpoint-url https://<handler_URL> --no-verify-ssl

Where:

• ./localfile.txt — the path to the file in the local file system.
• mybucket — bucket name;
• <handler_URL> — URL of the S3-compatible storage.

aws s3 rm s3://mybucket/bucketfile.txt --endpoint-url https://<handler_URL> --no-verify-ssl

Where:

• s3://mybucket/bucketfile.txt — key of the object in storage to be deleted;
• <handler_URL> — URL of the S3-compatible storage.

aws s3 ls s3://mybucket/ --endpoint-url https://<handler_URL> --no-verify-ssl

Where:

• s3://mybucket/ — the prefix in mybucket;
• <handler_URL> — URL of the S3-compatible storage.