5 reasons to claim 10TB of free object storage in City Cloud

Özgür Bal City Cloud, object storage Leave a Comment

Read the full article to learn how you can utilise OpenStack-based object storage to your advantage. Then claim 10TB of object storage in City Cloud, free throughout 2020, by simply registering an account.

Background

We are making a temporary change in our billing policy for object storage between 2020-06-03 and 2020-12-31. During this time, object storage usage will only be billed for storage space exceeding 10TB, effectively making the first 10TB free of charge for a limited time period. During this time object storage space exceeding 10TB per account will be billed as usual – per GB per hour according to our price list. Object storage billing will return to it’s normal state, whereby all allocated storage space will be charged, on 2021-01-01.

Jump to section

How object storage differs from other storage architectures

As opposed to storage architectures based on a file hierarchy such as NFS or blocks and sectors such as SAN, object storage is based on… you guessed it, objects. An object typically includes the data itself, some metadata and a globally unique identifier.

That last part, about a globally unique identifier, is one of the reasons that object storage is something that modern businesses needs to investigate in order to future proof their applications. You see, object storage was designed for cloud based IT infrastructure and can be implemented in both devices, systems and interfaces. Best of all, It’s programmable. So what this all means is that interfaces can be directly programmed by the applications or that a namespace can spam multiple instances of physical hardware and many other benefits.

What you’ll end up with is the ability to programatically retain massive amounts of unstructured data, wether you’re using object storage for your data lakes, disaster recovery or other business applications.

Why programmable matters

Programmable infrastructure is your VIP ticket in the digitalisation journey. Just like your IT infrastructure should and can be programmed through various orchestration and automation tools, your storage should to. Object storage allows you to do just that, connect your data sources and automatically store large volumes of data. In the case of object storage in City Cloud, it supports both the AWS S3 API and the OpenStack Swift API.

5 reasons to use object storage

Object storage can be used in a variety of ways to improve your business. Let’s look at 5 tangible cases and reasons to use object storage for your business applications.

1. Data lakes

Data lakes is one of the most hyped terms in the datasphere these days and this comes as no surprise. If you are ready to put all your data to work in order to improve your business value, a data lake is probably what you should have in mind. The inherent ability to store any kind of data, from multiple sources, in one single storage plane and without their respective silos is what makes object storage perfect for data lakes. Simply put, you can store any data that can be automatically fed to your data lake, analyse it and get comprehensive insights to improve your business.

Examples

Open source tools and frameworks for data lakes

https://www.logicalclocks.com/

https://hadoop.apache.org/

https://spark.apache.org/

https://cassandra.apache.org/

https://kylo.io/

2. Disaster recovery

Object storage is not all about data lakes and Big Data analysis. It is the perfect solution for backup and disaster recovery as well, mainly thanks to two things – distribution and cost.

Object storage is, at the core, very tolerant to failures and faults. It is after all a distributed storage system, built in a way that distributes your data across multiple storage systems in different physical locations automatically. Another aspect that increases its tolerance is the built-in ability to handle any kind of data, whether it be structured, semi-structured or unstructured. Just think about the client side in this equation. Your entire organisation can use one single storage pool to store anything from any device, even your backups. And getting back to distribution, even if a server or storage device fails, that in it self can have no impact on your data since the data is already available elsewhere.

Costwise, object storage is quite simply much cheaper than for instance block storage. The reason for this has to do with what different storage architectures are designed to do and how they are built. Block storage is optimised for performance and speed since it is designed to contain data such as the operating system for your VM’s or your databases. Object storage is optimised for cost efficiency and scale since it is designed to store large amounts of various types of data.

Open Source tools 

https://rclone.org/

3. Business applications

The scalability of object storage is yet to be matched by conventional storage techniques. Running out of space in an object storage system is a matter of adding more drives with relative ease. Combined with the programmable nature of object storage it is the perfect solution for online collaboration services and content platforms alike.

Just think about it like this. An object storage solution will give you one single storage plane for all your data. It is automatically replicated across multiple locations and scales very well. All you need to do is to connect your data sources, from anywhere where there is an API, and automatically store large volumes of data.

Open Source tools

https://rclone.org/

https://cyberduck.io/

4. It’s cheaper

Compared to other storage architectures, object storage is purposefully built to lower the total cost of your storage pool. Unlike for instance block storage, object storage is not optimised for speed but rather for cost per GB. A perfect example of exactly how much cheaper object storage is, can be found when comparing the storage solutions available in City Cloud. Block storage which is designed to handle operating systems for your VM’s, databases and other types of data which relies on performance and speed costs $0.00016 per GB per hour. Object storage, built solely for storing information and relying less on instant or real-time access to your data costs $0.0000416 per GB per hour.

5. It’s the cloud way of doing things

Running a modern business in this digital era comes down to two things. Speed and agility. Keep your speed of development up and prepare to pivot fast or you will risk losing your customers to a competitor that can satisfy their needs when they need it. This is true for virtually any business. Just think about it, how quickly could you open up a bank account 10-15 years ago? How easy was it to switch insurance companies? How fast do you abandon your cart if the online retailer can’t satisfy your shopping experience right this second?

Everything moves faster and businesses must too.

Thankfully there are a wide range of solutions, tools and methods to cope with these challenges. You’ve certainly heard about CI/CD and similar expressions to, at the core, explain the same thing – churn out digital solutions faster.

IT infrastructure, being the foundation of all things digital, has a particular buzzword of its own – programmable. Programmable IT infrastructure helps businesses automate and orchestrate their digital environments so that an army of programmers, network engineers and other developers doesn’t have to re-build everything every time a new idea needs to be tested and put in to production. Instead entire systems with everything from virtual machines and routers to storage and load balancing can be orchestrated with code. Then, when you need such a system for that brand new tool you are launching, just run the code and you are good to go.

This is the cloud way of doing things and object storage fits perfectly in to this method. 

Get started with object storage in City Cloud

Let’s look at how you can get started with object storage in City Cloud and get 10TB of free usage to test it out properly throughout 2020.

Create an account

The only thing you need to do is to register an account. Head on over to https://admin.citycloud.com/login and fill out your details to get started.

Policy

We are making a temporary change in our billing policy for object storage between 2020-06-03 and 2020-12-31. During this time, object storage usage will only be billed for storage space exceeding 10TB, effectively making the first 10TB free of charge for a limited time period. During this time object storage space exceeding 10TB per account will be billed as usual – per GB per hour according to our price list. Object storage billing will return to it’s normal state, whereby all allocated storage space will be charged, on 2021-01-01.

To ensure good user experience, and to protect our assets and systems, we reserve the right to introduce limitations to these temporary changes at any time. Limitations may include any measures to prevent bad user experience, misuse, overload, system failure or any other problem that could occur during this period. We will never claim payment for resources allocated prior to the introduction of potential limitations in relation to this temporary price change.

Setting up object storage in City Cloud

You’ll find the actual instructions below but first we want to give credit where credit is due.

Our fantastic education team provides a variety of educational services as part of our service portfolio. They are the ones who can help your organisation with both on and off-site training, labs, technical documentation and tailored cloud training for your specific needs. With our fully integrated cloud training platform, City Cloud Academy, we can help your organisation reach it’s true potential efficiently through self-paced online training.

As it so happens, we provide a self paced online course called OpenStack API and CLI Fundamentals. It covers all the basic knowledge you need to get started with OpenStack in City Cloud and in the course material, there is a separate section for object storage with OpenStack Swift.

If you are already familiar with the OpenStack API and CLI and just need to brush up on the particulars for object storage, we have provided those chapters in the course material for free below. If you need to read more about the OpenStack API and perhaps also get access to the labs, you need to enroll for the complete course. Contact us if you would like to know more.

RESTful Object Storage in OpenStack

In an OpenStack environment, you will typically deal with two types of managed storage.

  1. Block storage
  2. Object storage

You’ve already learned about block storage in the lesson on OpenStack Cinder. Typically for block storage, you interact with it through your virtual machines: Cinder volumes are attached to your VMs, they look like a regular block device, and you normally create a filesystem on them.

Object storage, more precisely RESTful object storage, is an entirely different storage pattern. Clients don’t interact with object storage using blocks or files. Rather, they use web protocols like HTTP and HTTPS, and the rich set of methods that these protocols provide, to interact with data.

REST

It is called RESTful because clients use Representational State Transfer (REST) to read, write, and modify data — exactly as they do when interacting with functional OpenStack services like Nova, Glance, Cinder, or Neutron.

This concept is not particularly novel, nor is it unique to OpenStack: the first widely-used RESTful object storage solution was the Amazon Simple Storage Service (Amazon S3), introduced in 2006. It is and remains a very popular object storage platform.

In OpenStack, the equivalent to Amazon S3 is OpenStack Swift.

Objects

An object is simply a chunk of binary data that lives in the object store. An object has a name, a certain size, some content, and optionally a number of arbitrarily named key-value pairs that we collectively call metadata.

Objects are organized in a particular way, which is somewhat distinct from how files are stored in a filesystem.

Containers

In Swift, objects are organized in containers (equivalent to buckets in S3). Any container can hold an arbitrary number of objects, but not other containers. That is to say, the container namespace is flat, rather than hierarchical, which is an important distinction from the hierarchical directory structure that typically comprises a filesystem.

Just like in a filesystem’s directory, object names must be unique within the container. However, unlike in a filesystem, / (the forward slash) is an allowed character in an object name, since in the absence of directories it obviously does not constitute a directory separator.

Containers, like their constituent objects, can also be assigned arbitrary metadata key-value pairs.

Service endpoint

Just like any other OpenStack service, Swift supports service discovery via the Keystone service catalog. The service is identified by the object-store service type.

Interacting with OpenStack Object Storage

To programmatically interact with OpenStack object storage, you use the appropriate API client library (which depends on the programming language you use). For Python, for example, you would use the python-swiftclient library.

However, in most circumstances you would initially not use the API and write your own program, but instead use a command-line interface (CLI) — either interactively from a shell prompt, or in a script.

Discovering the object store endpoint

As with all other OpenStack services, the object store endpoint is listed in the Keystone service catalog. You can discover it, using the openstack client, with the openstack catalog show object-store command.

Most of the time when you interact with object storage, you won’t need to manually do this: the object storage client will take care of the discovery on your behalf.

Using swift

The swift utility comes bundled with the python-swiftclient library. You can use it to interact with the object store, and its command line syntax is sometimes a little more compact than that of the generic openstack client.

For example, you can use swift list to list all your containers, or swift list foo to list all the objects in the container named foo.

You can also use swift upload and swift download to upload and download objects, and swift stat and swift post to display and modify container and object metadata.

Using openstack

If you prefer to use the generic openstack client, you can use that to interact with your object store as well, of course. The subcommands you’ll be looking at are openstack container and openstack object.

Here are the swift commands paired with their openstack equivalents:

Actionswift commandopenstack command
List containersswift listopenstack container list
List objects in containerswift list <container>openstack object list <container>
Create containerswift post <container>openstack container create <container>
Create objectswift uploadopenstack object create
Download objectswift download <container> <object>openstack object save <container> <object>
Download all objects in containerswift download <container>openstack container save <container>
Show metadata on objectswift stat <container> <object>openstack object show <container> <object>
Show metadata on containerswift stat <container>openstack container show <container>
Set metadata on objectswift post <container> <object>openstack object set <container> <object>
Set metadata on containerswift post <container> <object>openstack container set <container>

Using Third-Party Object Storage Clients

In the previous lesson, we covered the native OpenStack clients, openstack and swift. However, since the Swift API is open and popular, there are a number of third-party clients that also support the protocol, and that you can use to interact with your object store.

Rclone

Rclone is a very popular command-line application that allows you to synchronize files and directories from a multitude of local and cloud backends, including Swift. Rclone is written in Go and is open source, governed by the MIT License.

Cyberduck

Cyberduck is a file transfer client for Mac and Windows from Iterate in Switzerland, written in Java and GPLv3-licensed. It has full Swift support, and its documentation about how to connect to a variety of OpenStack service providers is quite extensive.

CloudBerry Explorer

CloudBerry Explorer is a freemium storage manager/browser for Windows that supports Swift.