Hadoop - the Big Elephant, IBM Smart Cloud and Windows Azure

According to my knowledge the two biggies who can take cloud to next level based on their market presence are Microsoft and IBM. Microsoft is already making big investments and has a clear cut vision on its cloud presence. However IBM is targeting small- and medium-sized businesses with a suite of cloud-based ondemand services via its Smart Cloud initiative. Although i haven't invested much time on IBM Smart Cloud, reading the market trend i feel IBM Is far behind from Microsoft.

IBM Smart Cloud: a series of cloud computing offerings that enables enterprises to distribute their computing needs across a globally accessible resource grid. One such offering is the Express Advantage suite, which includes data backup and recovery, email continuity and archiving, and data security functionality— some of the more data-intensive processes handled by a typical IT department. More information about this can be found here.

Now comes comes the question how Smart Cloud and Hadoop are related? IBM's Smart Cloud is set of tools used to manage IBM's hardware using Hadoop,which is based on the MapReduce software used by Google in its offerings. which is open source framework (I call it as Algorithms) for Elastic Scaling, Distributed File System and Data Storage Algorithms. Industry experts feel Hadoop is much more powerful than Azure Management services which is used to manage Azure Platform.

More about Hadoop can be found here: http://hadoop.apache.org

Where does Microsoft Azure Fit? SaaS or PaaS

One of the question which was doing buzz few years back was, under which category Microsoft Azure falls? SasS or PaaS. I strongly believe the first answer which comes to everyone's mind would be PaaS, let us understand the question itself first with Definition.

Software as a Service: Software as a service, or SaaS, is probably the most common type of cloud service development. With SaaS, a single application is delivered to thousands of users from the vendor’s servers. Customers don’t pay for owning the software; rather, they pay for using it. Users access an application via an API accessible over the web.

Each organization served by the vendor is called a tenant, and this type of arrangement is called a multitenant architecture. The vendor’s servers are virtually partitioned so that each organization works with a customized virtual application instance.

For customers, SaaS requires no upfront investment in servers or software licensing. For the application developer, there is only one application to maintain forb> multiple clients.

Now for Vendors, who develop SaaS kind of applications will require upfront investment on servers, Software licenses and they need to maintain and manage them. As they get more customers, their infrastructure has to change and they need to put more investment on people who manage this infrastructure. So it is most likely that vendor would take advantage of Microsoft Azure as Platform for hosting SaaS based application. In reality Azure will be the perfect place to put SaaS based application and it has Platform which can support both Multi Tenant and Single Tenant Applications.

Platform as a Service:In this variation of SaaS, the development environment is offered as a service. So Developer will be having ready made infrastructure to host their application. The developer uses the “building blocks” of the PaaS development environment to create his own custom application (Developer has to use Azure Blob objects to store stuff). As answered in the first place, Azure is the perfect place to use as PaaS.

Conclusion is Windows Azure fits into both Saas and Paas Architectures.

Elastic Scaling on Azure using Enterprise Library 5.0

One of the Biggest advantage and reason for moving towards Cloud (in specific to Windows Azure) is it gives customers Elastic Scaling of their Application instance. That is Customer can have the ability to Scale in and scale out anytime according to his business needs. However, doing this manually is pain and involves lot of monitoring and Automation with Management APIs. Azure platform provides Auto-Scaling using Enterprise Library 5, which includes a robust auto-scaling implementation that's ready to use out of the box.

Why Auto-Scaling?
Before thinking about Scale-in and Scale-out, one needs to define the metrics and understand the Business needs. There are probably thousands of variants, but the two most common types of metrics on which scaling decisions are based are:

1> Schedule (Timelines), such as particular weeks in the month: One of the example i think about configuring Auto-scaling based on this metrics is, during Christmas month on a eShopping site, we can always increase number of instances of servers.

2> Key Performance Indicators (KPIs), such as CPU utilization or queue depth.

How to get Metrics?
The next important question to answer is, how to get these metrics. For Schedule the answer is pretty simple, we all know when is the Christmas month, so we can take the date or time range and can configure auto-scaling based on that. How ever getting details about KPIs is not that easy, we need to develop a small monitoring tool which uses CPU performance counters using DiagnosticMonitor. Based on this data we can configure Auto-scaling configuration for KPI.

How Auto-scaling Enterprise Library works?
Auto-Scaling Enterprise library comes with a component called "the Autoscaler", which can be hosted as web role on azure or on the local box (on IIS 7). This tool needs to be configured with Management Certificate of Azure Account for which you are planning to Auto-scale (Azure Account for which you want to enable Auto-Scale).
Once you configured Azure account (Web role, Worker process or Azure Blobs), it is time to provide Mertics data which we collected to Autoscaler (We call this Autoscaler rules).

Once "the Autoscaler" is up and running, it can Automatically Scale-in or Scale-out based on the Rules you provided.

for more information on how to configure and install Autoscaler, visit the nuget gallery. http://nuget.org/packages/EnterpriseLibrary.WindowsAzure.Autoscaling

Azure Appfabric Cache: Difference between Partitioned Cache and Replicated Cache

Appfabric cache is something which is not new to distributed computing world. Microsoft released it as code named "Velocity" on Windows Server 2008 family few years back and Azure Appfabric caching is more like extension of this on Azure Platform.

What is Appfabric Cache?
Before getting into different types of Distributed Cache, let us quickly understand the definition of Distributed Cache.

distributed caching is a form of caching that allows the cache to span multiple servers so that it can grow in size and in transactional capacity. Distributed caching has become feasible now for a number of reasons. First, memory has become very cheap, and you can stuff computers with many gigabytes at throwaway prices. Second, network cards have become very fast, with 1Gbit now standard everywhere and 10Gbit gaining traction. Finally, unlike a database server, which usually requires a high-end machine, distributed caching works well on lower cost machines (like those used for Web servers), which allows you to add more machines easily.

Distributed caching is scalable because of the architecture it employs. It distributes its work across multiple servers but still gives you a logical view of a single cache. For application data, a distributed cache keeps a copy of a subset of the data in the database. This is meant to be a temporary store, which might mean hours, days or weeks. In a lot of situations, the data being used in an application does not need to be stored permanently. In ASP.NET, for example, session data is temporary and needed for maybe a few minutes to a few hours at most.

Difference between Partitioned Cache and Replicated Cache:
Appfabric Cache supports three types of caching techniques.
1>Partitioned Cache
2>Replicated Cache
3>Local Cache.

I'm going to ignore Local Cache in this topic as this is something which is known to computing world for ages now.

To understand the difference between Partitioned Cached and Replicated Cache. let us imagine Azure Platform having a Cache Custer with 10 Cache Hosts (Servers).
now if you have 100 CLR objects which needs to be cached on this Cache Cluster.

Partitioned Caching will store 10 CLR objects each on 10 Cache Host and Application can do Get and Put operation of Cluster and Cluster in tern picks cached data from the host on which it stored.

Advantage: Partitioned Cache can store huge amount of data as data is distributed across the hosts. I.e, if you have 1GB of Data to be cached and if the cluster has 10 hosts, then each host will be storing 100 MB of data. Put operation will be fast.

Disadvantage: If any of the host server goes down, then data will be lost. in general High Availability is not possible and GET operation is going to be slow.

Replicated Caching will store all the 100 objects on each of the 10 Cache host on the above scenario. In general each cache host will be having replica of all the Cache Data. I.e, if you have 1 GB of data to be cached and if the cluster has 10 host, then each host will be having complete 1 GB of data. This option provides high availability but it comes will lot of memory cost. Put option is going to be slow as data has to be synchronized across all the Hosts.

Difference between RESTFUL, Resource-Oriented Architecture Versus RPC-XML Sytle Architecuture

One of the learnings what i had in my previous project (Developing SDK for Restful Data Services) was understanding the difference between the above two Architectures. Although the difference looks very simple, understanding it at the core was always confusing. Here i'm going to list few key difference between these two architectures.

RESTFUL Resource-Oriented Architecture:
HTTP URL has everything which client is looking for, it has details about Web Method and Scope details. following example should give good idea about what i mean by Web Method and Scope.
Eg. http://www.google.com/search?q=TEST
in the above HTTP URL search is Web Method and Q=Test is scope detail (In other words Query String)

Always remember a thumb rule when it comes to RESTFUL, it is aways GET Request, which means Get HTTP URL (so URL has everything which client needs)

RPC-XML style Architecture
RPC-XML Architecture works on a thumb rule called POST Client Request to URL End point. So typle this approach has a HTTP End Point, Passing a Envelop full of data in the form of SOAP having Web Method and Scoping information in XML format and Recieving similar Envelop from client having response data in XML format.

In reality, developing this kind of Request Envelop and Response Envelop is developers nightmare, but cleaverly taken care by some proxy tools.

Factors to Consider before Deploying Application on Web Role

One of the first project real time assignment which we encountered during our Azure journey was moving Classic ASP website into Azure Platform with minimal or no code changes.  It was a kind of nightmare as our estimations got royally screwed and developers ended spending day and nights fixing all those classical session and COM integration related issues. However moving Classic ASP website to Azure might be an rare exception (refer my friend Girish's blog to understand more about this), i am going to list out factors which we need to consider while moving an Existing or new websites and web applications written using ASP.NET, MVC, Razor, or other .NET Framework technologies.

Distributed session state and output caching must be handled differently in an Azure Web Role from the typical process on-premises. You cannot use the ASP.NET State Server or the SQL Session State Provider. Instead, you configure an instance of the Windows Azure Cache Service and configure your application to use this, or use a third-party solution that saves data in Azure Storage instead.

 You must decide how you will store and access the data that the application uses. For simplicity, if an existing application uses a SQL database, you can upload the data into SQL Azure and access it in the same way as you would an on-premises SQL Server database. However, you may decide to use Azure Storage Tables or Blobs instead, which can prove less expensive but do not provide the same capabilities or performance as SQL Azure.

You cannot access the underlying operating system in the same way as you can when running code on a physical machine or in a VM Role. This prevents you from using COM components, accessing some system logs, and interacting with some operating system services.

If you depend on Windows authentication or use Active Directory for authorization (such as setting permissions on website folders and resources) you must rework the application to use ASP.NET Forms authentication and authorization, claims-based authentication, or a suitable custom implementation. Windows Azure Access Control Service (ACS) and Windows Identity Foundation (WIF) make it easy to implement federated claims-based authentication and authorization, and single sign-on. 

Unlike a web application running locally in IIS, changes to the Web.config file for a web application are not automatically applied at runtime. Instead you can place the settings in the Azure service configuration file so that changes are applied when you upload a new version of this file.

There is no local SMTP server for sending emails, though you can open port 25 for your Azure deployment and use an external email server.

You can configure a virtual NTFS volume to use in a similar way to a hard disk drive, but consider using other approaches such as Azure Blob storage for images, resources, and downloadable files. You can maximize the user experience and reduce load on your application by locating images, resources, and downloadable files on the Windows Azure Content Delivery Network (CDN).

Quick Introduction to Windows Azure Services and their Features

Lot of the content you see here is what i learned during my first training session during those initial Azure Days. Make a note that still lot of azure features are either beta or at pre-release stage and i have used September 2011 release of Azure SDK and tools while writing this blog. The services and features available change as Windows Azure continues to evolve.

The range of services and features available with Windows Azure, the Windows Azure .NET Libraries, and SQL Azure targets specific requirements for your applications. When you subscribe to Windows Azure you can choose which of the features you require, and you pay only for the features you use. You can add and remove features from your subscription whenever you wish. Again while i write this blog, i have 90 days of trail use of Azure services which really gives lot of insight on "how to use" part! 

I will split this introduction into four categories for better understanding as Execution Environment, Data Management, Networking Services, and other services. 

Execution Environment

The Windows Azure execution environment provides a platform for applications and services hosted within one or more roles. The types of roles you can implement in Windows Azure are:

Azure Compute (Web and Worker Roles). A Windows Azure application consists of one or more hosted roles running within the Azure data centers. Typically there will be at least one Web role that is exposed for access by users of the application. Just for a moment think that Web Role as IIS 7.5 instance available to you on Cloud. The application may contain additional roles, including Worker roles that are typically used to perform background processing and support tasks for Web roles. For more detailed information see "Overview of Creating a Hosted Service for Windows Azure" at http://technet.microsoft.com/en-au/library/gg432976.aspx and "Building an Application that Runs in a Hosted Service" at http://technet.microsoft.com/en-au/library/hh180152.aspx.

Virtual Machine (VM role). This role allows you to host your own custom instance of the Windows Server 2008 R2 Enterprise or Windows Server 2008 R2 Standard operating system within a Windows Azure data center. For more detailed information see "Creating Applications by Using a VM Role in Windows Azure" 

Data Management

Windows Azure, SQL Azure, and the associated services provide opportunities for storing and managing data in a range of ways. The following data management services and features are available:

Azure Storage. 

Azure Storage. This provides four core services for persistent and durable data storage in the cloud. The services support a REST interface that can be accessed from within Azure-hosted or on-premises (remote) applications. For information about the REST API, see "Windows Azure Storage Services REST API Reference"  The four storage services are: 

The Azure Table Service provides a table-structured storage mechanism based on the familiar rows and columns format, and supports queries for managing the data. It is primarily aimed at scenarios where large volumes of data must be stored, while being easy to access and update. For more detailed information see "Table Service Concepts" at http://msdn.microsoft.com/en-us/library/dd179463.aspx and "Table Service REST API" at http://msdn.microsoft.com/en-us/library/dd179423.aspx.

The Binary Large Object (BLOB) Service provides a series of containers aimed at storing text or binary data. It provides both Block BLOB containers for streaming data, and Page BLOB containers for random read/write operations. For more detailed information see "Understanding Block Blobs and Page Blobs" at http://msdn.microsoft.com/en-us/library/ee691964.aspx and "Blob Service REST API" at http://msdn.microsoft.com/en-us/library/dd135733.aspx.

The Queue Service provides a mechanism for reliable, persistent messaging between role instances, such as between a Web role and a Worker role. For more detailed information see "Queue Service Concepts" at http://msdn.microsoft.com/en-us/library/dd179353.aspx and "Queue Service REST API" at http://msdn.microsoft.com/en-us/library/dd179363.aspx.

Windows Azure Drives provide a mechanism for applications to mount a single volume NTFS VHD as a Page BLOB, and upload and download VHDs via the BLOB. For more detailed information see "Windows Azure Drive" (PDF) at http://go.microsoft.com/?linkid=9710117.

SQL Azure Database. 

This is a highly available and scalable cloud database service built on SQL Server technologies, and supports the familiar T-SQL based relational database model. It can be used with applications hosted in Windows Azure, and with other applications running on-premises or hosted elsewhere. It supports federation capabilities to make multi-tenant data storage and distributed storage easier to implement and manage. For more detailed information see "SQL Azure Database" at http://msdn.microsoft.com/en-us/library/ee336279.aspx.

Data Synchronization. SQL Azure Data Sync is a cloud-based data synchronization service built on Microsoft Sync Framework technologies. It provides bi-directional data synchronization and data management capabilities allowing data to be easily shared between multiple SQL Azure databases and between on-premises and SQL Azure databases. For more detailed information see "Microsoft Sync Framework Developer Center" at http://msdn.microsoft.com/en-us/sync.

 Caching. This service provides a distributed, in-memory, low latency and high throughput application cache service that requires no installation or management, and dynamically increases and decreases the cache size automatically as required. It can be used to cache application data, ASP.NET session state information, and for ASP.NET page output caching. For more detailed information see "Caching Service (Windows Azure)" at http://msdn.microsoft.com/en-us/library/gg278356.aspx.

Networking Services

 Windows Azure provides several networking services that you can take advantage of to maximize performance, implement authentication, and improve manageability of your hosted applications. These services include the following:

Content Delivery Network (CDN). The CDN allows you to cache publicly available static data for applications at strategic locations that are closer (in network delivery terms) to end users. The CDN uses a number of data centers at many locations around the world, which store the data in BLOB storage that has anonymous access. These do not need to be locations where the application is actually running. For more detailed information see "Delivering High-Bandwidth Content with the Windows Azure CDN" at http://msdn.microsoft.com/en-us/library/ee795176.aspx.

Azure Connect. This service allows you to configure roles of an application running in Windows Azure and computers on your on-premises network so that they appear to be on the same network. It uses a software agent running on the on-premises computer to establish an IPsec-protected connection to the Windows Azure roles in the cloud, and provides the capability to administer, manage, monitor, and debug the roles directly. For more detailed information see "Connecting Local Computers to Windows Azure Roles" at http://msdn.microsoft.com/en-us/library/gg433122.aspx.

Traffic Manager. This is a service that allows you to set up request redirection and load balancing based on three different methods. Typically you will use Traffic Manager to maximize performance by redirecting requests from users to the instance in the closest data center using the Performance method. Alternative load balancing methods available are Failover and Round Robin. For more detailed information see "Windows Azure Traffic Manager" at http://msdn.microsoft.com/en-us/WAZPlatformTrainingCourse_WindowsAzureTrafficManager.

Access Control. This is a standards-based service for identity and access control that makes use of a range of identity providers (IdPs) that can authenticate users. ACS acts as a Security Token Service
(STS), or token issuer, and makes it easier to take advantage of federation authentication techniques where user identity is validated in a realm or domain other than that in which the application resides. An example is controlling user access based on an identity verified by an identity provider such as Windows Live ID or Google. For more detailed information see "Access Control Service 2.0" at http://msdn.microsoft.com/en-us/library/gg429786.aspx and "Claims Based Identity & Access Control Guide" at http://claimsid.codeplex.com/.

Service Bus. This provides a secure messaging and data flow capability for distributed and hybrid applications, such as communication between Windows Azure hosted applications and on-premises applications and services, without requiring complex firewall and security infrastructures. It can use a range of communication and messaging protocols and patterns to provide delivery assurance, reliable messaging; can scale to accommodate varying loads; and can be integrated with on-premises BizTalk Server artifacts. For more detailed information see "Service Bus" at http://msdn.microsoft.com/en-us/library/ee732537.aspx.

Other Services

SQL Azure Reporting (Business Intelligence Reporting).

This service allows you to develop and deploy business operational reports generated from data stored in a SQL Azure database to the cloud. It is built upon the same technologies as SQL Server Reporting Services, and lets you uses familiar tools to generate reports. Reports can be easily accessed through the Windows Azure Management Portal, through a web browser, or directly from within your Windows Azure and on-premises applications. For more detailed information see "SQL Azure Reporting" at http://msdn.microsoft.com/en-us/library/gg430130.aspx.

Marketplace. This is an online facility where developers can share, find, buy, and sell building block components, training, service templates, premium data sets, and finished services and applications needed to build Windows Azure applications. For more detailed information see "Windows Azure Marketplace Data Market" at http://msdn.microsoft.com/en-us/library/gg315539.aspx and "Windows Azure Marketplace" (AppMarket) at http://windowsazure.pinpoint.microsoft.com/en-US/Default.aspx.