Private Clouds Are Here to Stay—Especially for Data Warehousing

Some cloud experts are proclaiming private clouds “are false clouds”, or that the term was conveniently conjured to support vendor solutions. There are other analysts willing to hedge their bets by proclaiming that private clouds are a good solution for the next 3-5 years until public clouds mature.  I don’t believe it. Private clouds are here to stay (especially for data warehousing)—let me tell you why.

For starters, let’s define public vs. private cloud computing.  NIST and others do a pretty good job of defining public clouds and their attributes.  They are remote computing services that are typically elastic, scalable, use internet technologies, self-service, metered by use and more.  Private clouds, on the other hand are proprietary and typically behind the corporate firewall. And they frequently share most of the characteristics of public clouds.

However, there is one significant difference between the two cloud delivery models –public clouds are usually multi-tenant (i.e. shared with other entities/corporations/enterprises). Private clouds are typically dedicated to a single enterprise – i.e. not shared with other firms. I realize the above definitions are not accepted by all cloud experts, but they’re common enough to set a foundation for the rest of the discussion.

With the definition that private clouds equate to a dedicated environment for a single or common enterprise, it’s easy to see why they’ll stick around—especially for data warehousing workloads.

First, there’s the issue of security. No matter how “locked down” or secure a public cloud environment is said to be, there’s always going to be an issue of trust that will need to be overcome by contracts and/or SLAs (and possibly penalties for breaches).  Enterprises will have to trust that their data is safe and secure—especially if they plan on putting their most sensitive data (e.g. HR, financial, portfolio positions, healthcare and more) in the public cloud.

Second, there’s an issue of performance for analytics.  Data warehousing requirements such as high availability, mixed workload management, near real-time data loads and complex query execution are not easily managed or deployed using public cloud computing models. By contrast, private clouds for data warehousing offer higher performance and predictable service levels expected by today’s business users. There are myriad other reasons why public clouds aren’t ideal for data warehousing workloads and analyst Mark Madsen does a great job of explaining them in this whitepaper.

Third, in the multi-tenant environment of public cloud computing, there is increasing complexity which will lead to more cloud breakdowns. In a public cloud environment there are lots of moving pieces and parts interacting with each other (not necessarily in a linear fashion) within any given timeframe. These environments can be complex and tightly coupled where failures in one area easily cascade to others. For data warehousing customers with high availability requirements public clouds have a long way to go.  And the almost monthly “cloud breakdown” stories blasted throughout the internet aren’t helping their cause.

Finally, there’s the issue of control. Corporate IT shops are mostly accustomed to having control over their own IT environments. In terms of flexibly outsourcing some IT capabilities (which is what public cloud computing really is), IT is effectively giving up some/all control over their hardware and possibly software.  When there are issues and/or failures, IT is relegated to opening up a trouble ticket and waiting for a third party provider to remedy the situation (usually within a predefined SLA).  In times of harmony and moderation, this approach is all well and good. But when the inevitable hiccup or breakdown happens, it’s a helpless feeling to be at the mercy of another provider.

When embarking on a public cloud computing endeavor, a company or enterprise is effectively tying their fate to another provider for specific IT functions and/or processes.   Key questions to consider are:

• How much performance do I need?
• What data do I trust in the cloud?
• How much control am I willing to give up?
• How much risk am I willing to accept?
• Do I trust this provider?

There are many reasons why moving workloads to the public cloud makes sense, and in fact your end-state will likely be a combination of public and private clouds.  But you’ll only want to consider public cloud after you carefully think about the above questions.

And inevitably, once answers to these questions are known, you’ll also conclude private clouds are here to stay.

 

Can Big Data Analytics Solve “Too Big to Fail” Banking Complexity?

Despite investing millions upon millions of dollars in information technology systems, analytical modeling and PhD talent sourced from the best universities, global banks still have difficulty understanding their own business operations and investment risks, much less complex financial markets. Can “Big Data” technologies such as MapReduce/Hadoop, or even more mature technologies like BI/Data Warehousing help banks make better sense of their own complex internal systems and processes, much less tangled and interdependent global financial markets?

Courtesy of Flickr

British physicist and cosmologist, Stephen Hawking, in 2000 said; “I think the next century will be the century of complexity.” He wasn’t kidding.

While Hawking was surely speaking of science and technology, it’s of little doubt he’d also look at global financial markets and financial players (hedge funds, banks, institutional and individual investors and more) as a very complex system.

With hundreds of millions of hidden connections and interdependencies, hundreds of thousands of various hard-to-understand financial products, and millions if not billions of “actors” each with their own agenda, global financial markets are the perfect example of extreme complexity.  In fact, the global financial system is so complex that even attempts to analytically model and predict markets may have worked for a point in time, but ultimately failed to help companies manage their investment risks.

Some argue that complexity in markets might be deciphered through better reporting and transparency.  If every financial firm were required to provide deeper transparency into their positions, transactions, and contracts, then might it be possible for regulators to more thoroughly police markets?

Financial Times writer Gillian Tett has been reading the published work of Professor Henry Hu at University of Texas.  In Tett’s article; “How ‘too big to fail’ banks have become ‘too complex to exist’ (registration required)” she says that Professor Hu argues technological advances and financial innovation (i.e. derivatives) have made financial instruments and flows too difficult to map. Moreover, Hu believes financial intermediaries themselves are so complex that they’ll continually have difficulty making sense of shifting markets.

Is a “too big to fail” situation exacerbated by a “too complex to exist” problem? And can technological advances such as further adoption of MapReduce or Hadoop platforms be considered a potential savior?  Hu seems to believe that supercomputers and more raw economic data might be one way to better understand complex financial markets.

However, even if massive data sets can be better searched, counted, aggregated and reported with MapReduce/Hadoop platforms, superior cognitive skills are necessary to make sense of outputs and then make recommendations and/or take actions based on findings. This kind of talent is in short supply.

It’s even highly likely the scope of complexity in financial markets is beyond today’s technology to compute, sort and analyze. And if that supposition is true, should next steps be to take measures to moderate if not minimize additional complexity?

Questions:

• Are “Big Data” analytics the savior to mapping complex and global financial flows?
• Is the global financial system—with its billions of relationships and interdependencies—past the point of understanding and prediction with mathematics and today’s compute power?

Top Financial Risks of Cloud Computing

Cloud computing definitely has upside as adopters can speed delivery of analytics, gain flexibility in deployments and costs, and transfer IT headaches to another company. However, with all the advantages of cloud, it’s important to keep in mind there are financial risks to cloud computing including potential costs from lawsuits and reputational damage from cloud provider security/privacy data breaches, and possible revenue losses from cloud provider downtime/outages.

For any type of business decision, there are various risks that should be considered– strategic, operational, financial, compliance and reputational (brand).  These risks should also be criteria for any decision to move workloads to cloud computing. However, for sake of discussion, let’s focus on financial risk.

First, for cloud computing there are financial risks in terms of potential data or privacy loss, especially in complex multi-tenant environments.  If there are data breaches of unencrypted personally identifiable information (PII), many US states have laws that require consumer notification. Companies accused of data breach also typically provide consumer credit monitoring services for up to one year. One research firm estimates total costs due to a data breach average $7.2 million (USD).  In addition, such breaches may open up companies to class action lawsuits that could total millions more in damages.

To mitigate risks of data loss or privacy breach, cloud providers do everything in their power to safeguard data including: server hardening, user provisioning and access controls, enforcement of policies for passwords and data privacy, monitoring/logging for intrusion detection, self-auditing, third party security audits (when specified), mandatory training for personnel and in some cases encryption of tables and/or columns.

And while in many cases the above practices are more robust in public cloud computing environments than in most corporate data centers, there are still lagging trust concerns of possible cloud data loss or privacy breach. Perhaps this is why, at least for the next 2-3 years, companies will increasingly choose private cloud over public cloud environments.

To mitigate financial risks some companies seek indemnification where the cloud provider agrees to take on or share liability of security breach including costs associated a breach. However, cloud financial indemnifications are extremely rare, and even if offered, the risk associated with such breaches is often transferred to insurance companies via purchase of cyber insurance. And of course, such insurance costs will be baked into cloud service fees.

Other financial risks for companies doing business in the cloud include loss of revenues if there are significant availability issues. If cloud environments are down for hours or days, this could adversely impact a business’ ability to perform analytics or reporting and thus may affect revenue opportunities. To offset possible lost revenues, most cloud providers will sign up for availability SLAs and associated penalties (usually redeemable as service credits).

Cloud computing has so much upside, that it’s very easy for business managers to declare “all things must be cloud”.  That’s well and good, but one must also carefully consider cloud risks. And while risk cannot be eliminated, it can surely be mitigated with proper planning and execution when things go wrong.

Companies considering cloud computing must remember that just like in outsourcing, there’s no such thing as transference of responsibility. In moving workloads to the cloud, carefully document upsides and downsides, examine your decisions in terms of risk (including financial ones), and then make the best decision possible for your particular organization.

Questions:

• This article speaks to financial risks for cloud computing in terms of access and availability. There are certainly more including project cost overruns for cloud deployment and data quality (completeness/accuracy).  What others can you think of?

Is Bigger, Better in the Cloud?

Bigger is better – is a phrase that’s widely assumed to be self-evident. However, whether it is cruise ships capsizing, or international banks catching a major cold/flu in the 2008 Financial Crisis, we know that while there’s presumably safety in the concept of “size” there can also be inherent complexity and subsequent risk.

Risk management is a critical topic business and IT professionals must take into account in terms of cloud computing.  And especially for mission critical data such as human resources, payroll, financial or even patient data, security and privacy of sensitive data is a paramount concern when considering cloud delivery models.

Courtesy of Flickr/Serhat Demir

But in cloud computing, risk comes in other forms as well including financial viability, especially when there seems to be a new cloud entrant in the marketplace every week. New and attractive markets usually attract entrants at a sizzling pace, however, when the eventual market shakeup comes due, there’s also a chance your cloud provider might go out of business completely, taking your data and applications with them.

And let’s not forget operational risk in the cloud, where it might be assumed large cloud providers might have the upper hand in hiring the talent and expertise necessary to manage inherently complex cloud environments.   However, all the talent in the world is not going to save an environment that’s poorly architected, tightly coupled, and one operational mistake (or bad decision) away from catastrophic meltdown.

Ultimately, one cannot master risk. Instead, management of risks is about all we can hope for.

Mark Twain once famously said; “put all your eggs in one basket – and then watch that basket.” However, Mr. Twain surely didn’t have cloud computing on the mind when he spoke.

For IT and business professionals considering cloud computing solutions, it’s probably tempting to short-list those providers that have a sizable and large cloud computing presence (e.g. the top ten largest and established vendors). However, for a few of these companies, cloud computing is an ancillary business and there’s no guarantee that strategic plans won’t shift to the point that spin-offs aren’t a possibility. In addition, with cloud computing margins already thinning by some estimates, there’s also a good chance investor pressures may force cloud providers to skimp on redundancy or recklessly cut corners elsewhere.

Long way of saying, when it comes to cloud computing, I’m not convinced there’s safety in numbers, nor that a bigger presence and/or market share signals a fundamentally better offer.

Questions:

• When it comes to cloud computing, do you believe bigger is better and possibly safer?
• What criteria do you look for in assessing a cloud provider?