Nuno Filipe Godinho : SQL Azure

How to choose the right Relational Database Service you should use in Windows Azure

NunoGodinho — Thu, 20 Sep 2012 21:51:19 GMT

Now with Windows Azure SQL Database as well as SQL Server inside a Windows Azure Virtual Machine an important question comes up, and that is, Which Relational Database Service should I use for my solution in Windows Azure?

In order to help answering this question I did a flowchart that should help. (this is a simplification of the process but should answer most of the questions)

What are the features that aren’t supported in SQL Database you might be asking. Here’s a list:

SQL Server Utility
SQL Server PowerShell Provider. PowerShell scripts can be run on an on-premise computer, however, and connect to Windows Azure SQL Database using supported objects (such as System Management Objects or Data-tier Applications Framework).
Master Data Services
Change Data Capture
Data Auditing
Data Compression
Extension of spatial types and methods through Common Language Runtime (CLR)
External Key Management / Extensible Key Management
FILESTREAM Data
Integrated Full-Text Search
Large User-Defined Aggregates (UDAs)
Large User-Defined Types (UDTs)
Performance Data Collection (Data Collector)
Policy-Based Management
Resource Governor
SQL Server Replication
Transparent Data Encryption
Common Language Runtime (CLR) and CLR User-Defined Types
Database Mirroring
Service Broker
Table Partitioning
Typed XML and XML indexing is not supported. The XML data type is supported by Windows Azure SQL Database.
Backup and Restore
Replication
Extended Stored Procedures
SQL Server Agent/Jobs

More about this Windows Azure SQL Database:

More about SQL Server inside a Windows Azure Virtual Machine:

Hope this helps you to make the right choice.

Tips & Tricks to Build Multi-Tenant Databases with SQL Databases

NunoGodinho — Sat, 11 Aug 2012 12:55:17 GMT

Last Wednesday I delivered another session at the Visual Studio Live @ Redmond conference this time about “Tips & Tricks to Build Multi-Tenant Databases with SQL Databases”. The feedback from the session attendees was very good and this is a quick summary of the most important aspects.

First thing in order to be successful is understanding exactly what is Windows Azure SQL Databases (formally SQL Azure) and how it works. If we look at it from a high level, SQL Databases are actually:

SQL Server database technology delivered as a service on Windows Azure

This is actually a Shared Environment where we have SQL Server capabilities and features available in a pay as you go, and scalable mode. This of course doesn’t mean we have all the existing features and capabilities from SQL Server, since some of them would probably create some issues since this is a Shared Environment and also because the goal is High Availability and Scalability.

Ideal for both simple and complex applications

It’s a way for us to have a Relational Database as a Service quickly and powerful which can be used in all types of solutions, but in order to get the best out of it we really need to understand and adjust how it works and avoid things like being throttled, for example.

Enterprise-ready with automatic support for HA

True since it provide a higher level of management since we don’t deal with physical machines anymore but with the actual storing of data this way focusing more on what’s more important for our solution and our business.
Also this provides us an automatic support that enables us to have High Availability without ‘a lot of work’.

Designed to scale out elastically with demand

SQL Databases were created to scale and of course that’s part of it’s DNA, since without the ability to scale elastically it wouldn’t be fit for the Cloud. This will be allowed with SQL Federations.

After a high-level understanding of what is a SQL Database in Windows Azure, it’s important to understand also what are the scaling strategies that we can use, since this way we can better use them whenever needed. And so the strategies are:

Horizontal Partitioning or Sharding

Spreads data across similar nodes

All nodes have the same schema

Allows us to achieve massive scale out both in terms of Data Size and Load
We need to understand that while doing that we aren’t going to be able to get the complete list of data within a single query, and so it’s something we need to understand and consider.

Vertical Partitioning

Spreads data across Dis-Similar nodes

Each node has it’s own schema where the data is stored. Eg. SQL Database, Table Storage, Blob Storage, …

Allows us to place the data where it makes more sense by slitting the data we have for a solution and understanding how it’s used and how it should be stored so we can take the best out of it.
In this case we need to understand that by doing that we a splitting at the row level, so if we want a complete row (if we were to be thinking about a regular database) we won’t be able to get that in 1 only query, since one part can be in a DB, another in Table Storage and another in Blob Storage.

Hybrid Partitioning

This is when we spread our data both in Vertically and Horizontally.

Now let’s have a quick look at SQL Federations:

Integrated database sharding that can scale to hundreds of nodes

Provides the ability to do Horizontal Partitioning or Sharding to data inside SQL Databases in a quick and ‘easy’ way.

Multi-tenancy via flexible repartitioning

Provide the ability to achieve multi-tenancy inside the same Database by providing the ability to split data horizontally.

Online split operations to minimize downtime

Automatic data discovery regardless of changes in how data is partitioned

Finally, before we get to the actual Tips & Tricks we need to understand the multi-tenancy strategies that are typically used, and they are:

Separate Servers

This provides the best isolation possible and it’s regularly done On-Premises, but it’s also the one that doesn’t enable cutting costs, since each tenant has it’s own server, sql, license and so on.

Separate Databases

Very used in order to provide isolation for customer, because we can associate different logins, permissions and so on to each DB. Considered by many the only way to provide isolation for tenants.

Separate Schema

Also a very good way to achieve multi-tenancy but at the same time share some resources since everything it’s inside the same DB but the schemas used are different, one for each Tenant. That allows us to even customize a specific tenant without affecting others.

Row Isolation

Everything is shared in this option, Server, Database and even Schema. The only way they are differentiated is based on a TenantId or some other column that exists on the table level.

So now that we had a high-level view of all this let’s take a look at some of the Tips & Tricks for it, and they are:

Choose the right Multi-Tenancy Strategy

One of the most important steps for delivering a Multi-Tenant solution is understand exactly what should be the approach we should use, and normally the simplest one isn’t actually the best. For example, if we think about Isolation the best might seem Separate Server or Separate Database, but that means that from an economics standpoint we aren’t going to be very competitive, and so for this we need to understand that is we go further for a more shared approach, like Row Isolation, the impact in terms of development might be that at the beginning the investment is bigger, but in the long run that will pay off.
Also important is that if we want Multi-Tenancy and Isolation the only solution is not separating everything, since that is something we can enforce programmatically, through security permissions and so on. It’s just might take a bit more effort but the end result should be that we achieve some other customers that we usual didn’t because of the prices.

Don’t forget Security Patterns

Some time when we start creating Multi-Tenant Databases we start thinking about sharing everything and forget about the Security part of that. This is actually a common mistake that can cost a lot, and in order to make things work we should never forget to:

Filter

Use an intermediate layer that will receive all the requests to the DB and infer the Tenant filter, so that nobody has access to something that shouldn’t. Of course this also means that no one should have direct access to the DB.

Permissions

Very important when we consider Multi-Tenant Databases is the Permissions and how we can affect them. When we use Separate Servers, Databases or either Schemas, we can actually associate different logins, roles and so on to the different Tenants, but when we are in a Row Isolation model that isn’t possible, and that’s why the intermediate layer, that is actually your Data Layer will be very important since not only provides access to the data inferring the filter by Tenant, but also allows us to introduce permissions to access certain parts of the data. For example by leveraging Windows Azure Active Directory Access Control Service.

Encryption

This is of a huge importance. IF YOUR DATA IS SENSITIVE JUST ENCRYPT IT. SQL Databases don’t have the ‘With Encryption’ for Columns but this doesn’t mean I can’t really insert encrypted data in the database, I just need to do it on another layer, again in an intermediate layer.
Also very important when we encrypt our data is to understand the method we’re going to use. Normally one of the best methods, if we don’t want anyone that isn’t part of a Tenant to access the data and have different encryptions per Tenant so that even if someone gets the full DB it won’t access the full data, is to use X.509 Client Certificates. This is a very good approach since the Client is actually the one that has the Certificate that is used, but it also means that we cannot count of doing background calculation with that data in the Cloud since we don’t have the certificate. So it’s a balance.
A quick reminder is that IF YOU SHARE A SECRET IT STOPS BEING A SECRET, it’s just like telling a secret to a child. So for this reason, if you use X.509 Client Certificates to encrypt the data, and then register all those certificates in a Windows Azure Role, that isn’t the best approach because if someone get access to that role it will have access to the KEYS OF THE KINGDOM.

Choose your partitions wisely

When you choose the partitions it should be based on the ‘hotness’ of the data and not on the ‘# of records’. This is a very important premise since we normally see partitions being made in a way that they all have the same amount of records and data, and this doesn’t mean the solution will perform, because if we have one partition that has all the most commonly used data and another one with less common we won’t have any benefit with the partition. So the important part is to partition your data based on how the data is used and how commonly it is used, since in order to get good partitioning sometimes we can have a partition with very few records and another one with thousands.
So it’s also important to before partitioning the data understand how that data will be consumed and used, since that will allow us to better understand what is the most used data or not.

Choose your Partition Keys correctly

We can have several ways of defining partition keys and the most common are:

Natural Keys

Choosing a Natural key is usually one of the most used ways in order to select a partition key, and some samples are:

Tenant
Country
Region
Date
…

But sometimes this isn’t the best approach since if we go for the Tenant that doesn’t partition based on the ‘hotness’ just based on the Tenant and if a tenant is small has less information and so it will be faster than a tenant that is larger because it has more data and all in the same partition. This is exactly what we want to avoid. The same thing happens with Country, since really isn’t the best way since if we use this we might have more customers from a specific customer than from another, and the same with region. When we partition by data, it will mean that everybody would be affected by everyone else data, since everything would be in the same partition.
What this mean is that while Natural Keys are one of the most used partition keys they aren’t actually the best choice because it’s very difficult to find something that allows us to partition it optimally.

Mathematical

This is another option for partition key since what we can do is used things like Hash or Modulo operator and other options to generate a mathematical calculation in order to find the ‘hotness’ point.
Being this a very interesting approach is actually really difficult also since you need to understand your data very well as well as built mathematical formulas which isn’t everyone best hobby and capability.

Lookup Based

Another option is this one, Lookup based, this is actually the best since it really looks at how the data is used and consumed in order to find out the best partition key. In some cases this will mean a concatenation of something like ‘TenantId+Date’ or something like that, because in this case we’ll be saying that every tenant is partitioned independently and even at the same time is partitioning its own data making it faster.

Leverage SQL Federations to handle Multi-Tenancy

SQL Federations is a very good way to leverage Horizontal Partitioning (Sharding) of data for you solutions since allows us a ‘quick and easy’ way to partition our data and provide at the same time Isolation, since each Federation Member is actually a separate DB that is generated, but when we look at it we only see the main DB.
Currently the limitations with this is actually the fact the SQL Federations only allow the partition key to be of types BigInt, Int, UniqueIdentifier and VarBinary. It would be great if it would support also Varchar but we can’t have all, unfortunately, but if we go for a partition key like ‘TenantId+Date’ this can be actually a number and so fall inside the BigInt possibility.

Don’t forget that you’ll continue to need to perform backups

Sometimes when we are in a multi-tenant environment we forget we still need to provide backups and not only for us, but most of the times the customer wants also to have a copy of the data and that is more challenging.
In order to do that backups we can use the Export capability from SQL Databases as well as some third-party tools like RedGate’s SQL Azure Backup.
To leverage a backup of the data for the customer we can leverage SQL Data Sync since it allows us to create filters in the data sets and so we can filter by tenant and get one Data Sync per Tenant this way making everyone happy.

So those are some of the Tips & Tricks you can use in order to be successful building Multi-Tenant Databases in Windows Azure SQL Databases. I hope that helps and would love to have your thoughts about it.

Visual Studio Live @ Las Vegas Presentations – Architecture Best Practices in Windows Azure

NunoGodinho — Thu, 29 Mar 2012 16:50:00 GMT

Unfortunately I wasn’t able to go and speak in Visual Studio Live @ Las Vegas as it was scheduled, due to an illness that made it impossible for me to travel, and stay in bed for a few days.

But even if I wasn’t there I would like to share with you some of the points on this topic “Architecture Best Practices in Windows Azure”.

Architecture Best Practices on Windows Azure

View more presentations from Nuno Godinho.

Here are 10 key Architecture Best Practices in Windows Azure:

Architect for Scale
Plan for Disaster Recovery
Secure your Communications
Pick the right Compute size
Partition your Data
Instrument your Solution
Federate your Identity
Use Asynchronous and Reduce Coupling
Reduce Latency
Make Internal Communication Secure

In some next entries I’ll go in-depth into each one of those.

Visual Studio Live Orlando

NunoGodinho — Tue, 08 Nov 2011 10:38:04 GMT

This is a very interesting 5-day event on Microsoft Technologies that will be held in Orlando, and has tons of great sessions and speakers around topics like, Windows Azure, SQL Server, WCF, WF, Server AppFabric, MEF, Silverlight, WPF, Visual Studio, Team Foundation Server, ALM, VS11, HTML5, CSS3, ASP.NET MVC, Orchard, Windows Phone 7, JQuery and more.

Seems to be a really great event, that I’d recommend to anyone that wants a really powerful way to fast learn the topics, from beginners to advanced.

Check out more here.

Windows Azure Best Practices

NunoGodinho — Thu, 13 Oct 2011 18:17:00 GMT

I was on Twitter and suddenly some remember that it would be interesting to write some Best Practices about using Windows Azure. Here are some of the ones I remembered:

Get out of the way when you can. Send client directly to blob storage for static content
Use Shared Access Signatures to provide direct access to ACLed content. Can be time-bound and revoked on demand. Also important will be using a Shared Access Policy in order to remove information like end date from url
Serve public blobs from the edge with the Windows Azure CDN. You have fewer hops, and be closer to the customer.
If you're using #CDN don't forget to Manage the Content Expiration. The Default is 72 hours
If you're using CDN use versioned urls to allow immediate refresh. Really important to avoid caching problems
Cache hot data in memory to avoid slower data-tier access. Ex. Session State, immutable reference data. Using the Caching ties will help you reduce latency and in some cases also costs
Consider using more smaller compute instances instead of few large one. Most times you'll get better performance
Partition Data based on the indexing needs. Use #SQLAzure for highly indexed data, and #Storage for the rest. Really consider doing Hybrid Partitioning
Shard your #SQLAzure data across databases to increase the workload.
Don't forget to do the basic performance tuning to applications. Measure, optimize & create a baseline for performance
Don't forget to enable compression for additional dynamic content types in Web Apps
You can't update Apps with a different number of endpoints, so prepare for that upfront, leave some backdoors opened
In #Azure #AppFabric Caching Prefer to cache more smaller objects then few larger ones, increases your performance
Consider VMRole only when you need something installed that takes long time or is a non-automated install
Use Startup Tasks preferably in background unless you need a specific order in their launch
Use Asynchronous Work Pattern and Queues instead of using a Synchronous approach on your Roles
For devices don't connect directly to #SQLAzure, proxy those in your #Azure Compute (of course #SOA approach)
If you use Startup Tasks don't forget to log everything. Ex. command >> %~dp0log.txt 2>> %~dp0err.txt
Don’t forget to use “/y” in everything to avoid confirmation messages. Ex. “copy /y source destination”
For Startup Tasks use local storage since it’s a guaranteed space, and if there’s not enough space it won’t run.
If you need to run a startup task that needs the IIS pool to exist, then you should use OnStart() in elevated Role – This isn’t a problem since running a Role in Elevated mode doesn’t affect how the Web Application will run on IIS
Process that needs to be monitored you should use a ProgramEntryPoint or NetFxEntryPoint instead of Startup Tasks
Define NetworkTrafficRules in order to better protected internal access to your roles
If you’re using Traffic Manager build a AppHealth page Monitor that checks if everything is OK in your Application
When using Table Storage use the SaveChangesOptions.Batch to improve performance and save transactions
Choosing your VMSize is important, choose accordingly to your Speed, Memory, IO and Network needs.
Use Retry Logic for #SQLAzure connection and for #AppFabric Service Bus, Cache and Storage. Use Transient Fault Handling Framework for SQL Azure, Windows Azure Storage, Windows Azure AppFabric Service Bus and Cache.
Don't use owner for #AppFabric Service Bus. Create your own login and provide them the appropriate permissions. Check this post by Neil Mackenzie.
Don't use the #SQLAzure login that you created on the Management Portal in the connection String. Create your own. This is because you need to work in the least privileges and you don’t want to use the SA for that.
When using Queues instead of using GetMessage use GetMessages(count), this will help you save transactions and improve performance of your solution
If you're using Workers that are Queue based (Pooling Mechanism), use a back off mechanism when no messages are found in the queue. This will help reduce the number of transactions that are done and improve your solution performance
If you're using Workers that are Queue based (Pooling Mechanism), you should consider using notifications to resume the back off mechanism. #AppFabric Service Bus will be perfect for this, but don't forget to do the cost calculation to your scenario.
Perform your Application Scaling process considering all options, Scale Up, Out and Down, don't forget Down.

Last Updated 2011-10-21.

BUILD WINDOWS CONFERENCE 2011 – Keynote 2

NunoGodinho — Thu, 15 Sep 2011 00:57:16 GMT

Satya Nadella - President of Server and Tools Business @ Microsoft

Connected Devices and Continuous Services

Design Point

Elastic resources
Composable services
data as a namespace

Expose data as a first class namespace for other developers to use

identity & access
continuous delivery

This design point was the base for building Windows Azure and the Windows Server 8

Scenario: Building a Metro-Style Application with Jason Zander – Corporate VP, Visual Studio @ Microsoft

Visual Studio 2010 & Windows Azure Tools for Visual Studio – September

Managed Service Configurations

Allow you to create several configurations for the same cloud service, like Staging, QA, Production, and so on.

Advanced Diagnostics, including profiling is now available on Windows Azure
Visual Studio 11 has now the Productivity tools included in the base and not has an extension
Now available to search for Copy & Paste code, even small sections of the code
Visual Studio 11

Designer for DirectX also available with supports for effects, alpha blending, and more.
Diagnostics of DirectX viewing that gives you the frames that were rendered
Diagnostics from DirectX views included with call stack

Announcements:

Visual Studio 2011 Developer Preview
Windows Azure SDK 1.5
Windows Azure Toolkit for Windows 8

App Platform and Application LifeCycle Management (ALM)

Application Platform – Scott Guthrie – Corporate VP, Server & Tools Business @ Microsoft

ASP.NET MVC 4

new Style for the default site, and with security and more elements already wired up.
DOM Explorer also available for Web Apps
HTML Inspector is now available inside Visual Studio
Live Design to enable the Page Inspector as a different Pane that allows you to view the Page Inspector and the code side by side.
Bundeling and Minification

we reference a folder and it bundles on the server side all the contents of the folder and send it to the customer as a single and minified file

Async and away language capabilities available on C# and VB.NET

Enhances greatly the performance both in .NET 4.5 and Windows Server 8

There’s a new Mobile ASP.NET MVC template to create a new Mobile app, and also we can start using a already build ASP.NET MVC 4 project and mobile enhance it using:

jQuery.Mobile.MVC NuGet Package

jQuery.Mobile will shipped as now part of the ASP.NET and Visual Studio from now on.
This also adds a new file called _Layout.Mobile.cshtml

Will override the layout when the site is called by a mobile device

Supports iPhone, Windows Phone or Android

.NET 4.5 and Windows Server 8 support WebSockets deeply

WebSocket API in .NET 4.5 that allows us to connect to WebSocket from everywhere.

Deployment

Add Windows Azure Deployment Project is now available in every project type and will add immediately the Cloud Service project, configure and them deploy it to Windows Azure.

ALM – Jason Zander – Corporate VP, Visual Studio @ Microsoft

Hosted TFS Preview – Visual Studio Team Foundation Server in Windows Azure

SaaS – Software as a Service
Visual Studio 11 and ALM is using SCRUM Template
Easily add user stories, make the sprint planning, create tasks, Task Board in order to manage the stand up meetings
Works directly in Visual Studio 2010 with a specific patch

Announcements

ASP.NET MVC 4

Windows Server 8

High availability & management
Virtualization & cloud

Built on top of a high availability platform
Taken some parts of Windows Azure into this version

App Platform
Identity & secure access

claims-base identity

Note: There is a huge symmetry between Windows Azure and Windows Server
Capabilities – Bryon Surace – Senior Project Manager @ Microsoft

Storage

New File Services Manager
Storage Spaces
You don’t need a Phd in Storage, just connect the disks

Networking

Multiple High-Speed NICS, previously available only on HPC

Virtualization

Hyper-V supports now using a virtual disk from a file share
Now in Windows Server 8 we can move a disk from one machine to a remote file share without stopping anything.

Announcement

Windows Server 8 Developer Preview is already available for download

Windows Azure

High cadence of releases
Current features

Compute
Database
Storage
Virtual machines
CDN
Caching
Virtual Network
Service Bus
Marketplace
Business Intelligence
Access Control

Key services

Database Platform

Reporting
SQL Azure
SQL Server
Windows Azure Storage
SQL Data Sync

Data Marketplace

eBay uses Bing Translator DataSet from Data Marketplace to translate the texts

Access Control Service

Scenario: Build a Metro-Style application with Identity & Access Control using Windows Azure ACS – John Shewchuk – Technical Fellow @ Microsoft

How to Build the Margie’s Travel app with ACS

Using ACS and Windows 8 Security APIs
All of this code is available on the Windows Azure Toolkit for Windows 8

Windows Azure Announcements

Windows Azure Service Bus September release
Windows Azure Geo-Replication now enabled by default in Windows Azure Storage
Windows Azure Marketplace Expansion

Additional 25 Countries available
New Bing Data

Hybrid Scenario – Jeff Sandquist (Senior Director, Channel Nine @ Microsoft) and Dan Fernandez (Senior Director @ Microsoft)

Using Service Bus Messaging

Topics ( Reliable Messaging )

Connected Devices and Continuous Services

Opportunity

Windows Azure
Windows Server 8
Visual Studio 11

Personal Notes

This was a very interesting Keynote that showed up some very cool things, like Visual Studio 11 that looks very interesting, and very slick, Windows Server 8 that is really a Server OS that leverages the elasticity that we love in the cloud, and that can help a lot make our businesses better, but the things that really impressed me was the scale up and Hyper-V capabilities of it and that it was built to leverage and use the Windows Azure and Cloud lessons that we’ve learned and bring them into our company.

Of course if you know me I’m a more Windows Azure guy and so Windows Azure was really very impressive also, and mainly the geo-replication of Windows Azure Storage is huge, because we now have the capability of having our storage replicated in more than one Windows Data Center, and this is done automatically without any need of change. This is huge in terms of disaster recovery and security of your data, since in the end of the day we don’t want to lose anything. Also the usage of Windows Azure AppFabric Service Bus Messaging is very interesting and will enable us to make more reliable solutions that is very important. So I’d say, great job Microsoft and continue to invest on those and that leveraging between Windows Azure and Windows Server 8 is very interesting and I’ll dig more in order to see exactly how it will work.

Finally the new capability that Windows Azure brings to allow that any project can be changed into a Cloud project with a single selection in the context menu is very good since it allows us to really start taking more Windows Azure into account.

Things to take into account when you deliver applications for Connected Devices – Windows Azure to the rescue (EN)

NunoGodinho — Thu, 16 Jun 2011 21:43:29 GMT

Currently one very important space is the development of Applications for Connected Devices like Windows Phones, Windows Slates and so on.

One important elements is that we are currently going more and more to Wireless, and so it’s important to understand the current state of the art on Wireless.

According to AT&T “We will deliver as much data over our network in the first 5 weeks of 2015 as we did all year in 2010”.

Also currently there are several identified congestion factors that are affecting Wireless, like:

More subscribers with smartphones consuming data, uploading and downloading photos and videos
Spectrum + cell towers + antennas + home users
Mobile Operator bandwidth caps
Latency of all communications between the server and the destination device, passing by the Cell Towers, Backhaul network, internet, firewalls and so on

And so having this is mint, it’s really important to understand that we as developers cannot continue thinking that we’re going to continue developing our apps and services the same way, without thinking about these network related topics, because this will affect greatly the User Experience of our Solutions. And one important note is that UX is not only related to the User Interface (UI), like some developers try to think, it’s more then that. It’s really about having a great experience while using our applications.

So in order to do great applications for connected devices, and support millions of those devices, we need to think about scale.

Some examples of how we can scale our applications are:

Making Database that almost nobody touches, since we really shouldn’t let every single device connect and work on our databases, instead we should really be doing multi-tier application to abstract those devices from the Database, and by doing that we are reducing the stress of our databases and at the same time enabling the possibility of scaling the services that really give us access to our data
Place thousands of Web Servers on commodity hardware, by scaling out their app
Replicate Data
Scale horizontally
Use NoSQL databases at the edge
Shard our data using small tables for fast access
Place Load-Balancing at every tier
Use Hundreds of terabytes of data in an in-RAM distributed cache (For example Facebook uses MemCache and has something like 300 TB of data in memcache)
Use pre-compile elements, since compiling slow (For example Facebook uses PHP because they think that is very productive for them, but it didn’t work very well in terms of compiling, so they made a team to create a PHP to C++ compiler in order to get more performance)
use MapReduce pattern for parallel analysis of “Big Data”

Some of the apps that are doing this are, Bing, Facebook, Twitter, Google, Zynga, and so on.

So how in order to achieve that kind of scale we have Windows Azure. But what should we do to take the best out of Windows Azure for Connected Devices?

Scale-out. This is easy in Windows Azure, because we can define the number of instance we want of a particular role
SQL Azure should be never touched by the Devices. Instead we should use:
- In terms of Outbound Data - Worker roles that replicate outbound data from SQL Azure to NoSQL Azure Table Storage
- In terms of inbound Data - Worker Role move inbound data from Azure queues to SQL Azure
Use Distributed caching, that in Windows Azure is handled by AppFabric Caching
Do wireless efficient Services. For example using Web Roles that expose REST + JSON WCF Services is very interesting, because it saves message size, and by doing so saves time and money
Also important is making our devices cache data and work on it offline. Not every thing should be made “live” and on top of “live” data. Don’t stress the wireless connection

I hope this helps you think better about your Connected Device Apps.

MIX11 - Session Review - Powering Data on the Web and Beyond with SQL Azure

NunoGodinho — Fri, 22 Apr 2011 23:21:54 GMT

David Robinson (Senior Lead Program Manager)

SQL Azure Relational Database

Based on SQL Server 2008 R2 engine
Use same tools and data access frameworks
Six global datacenters
High Availability & Redundancy

Reads are completed at the primary
Writes are replicated to a quorum of secondaries

SQL Azure Roadmap

MIX08

Introduced SQL Server Data Services (ACE type)

MIX09

Introduce SQL Data Services (Real Relational Database)

MIX10

New Features announced

50 GB Database
PHP Support
MARS support
OData Endpoint Preview
Geo Spatial Data Types

ASP.NET MVC 3

Razor view Engine which provides a very streamlined syntax for writing clean and concise views.
Improved support for Dependency Injection
Global Action Filters
jQuery based Unobtrusive Ajax and Client Validation
ViewBag property for dynamic access to ViewData
Support for view engine selection in the New Project and Add View dialog

Extending the Reach of Data

This is achieved using ODATA protocol
Cloud Data Services

Windows Azure Tables
SQL Azure
Windows Azure MarketPlace DataMarket

Powering Web/AJAX Experiences

Consuming ODATA via jQuery & JSON

Powering Windows Phone 7

ODATA libraries available for Windows Phone 7

Powering Interoperability on Other platforms

Why does Performance matters?

More responsive applications
Faster page load times

8 seconds vs 3 seconds

Higher interactivity – new type of applications
Better User Experience – more $$$$

SQL Azure Data Sync

Sync within the Cloud
Read scale-out via multiple copies

reporting
web site reference data

Read-write scale-out via multiple copies:

ex. geo located web apps

Same or different data centers
Key features

No-Code sync configuration

Easily define data to be synchronized and locations
Choose how often data is synchronized

Full Data Synchronization Capabilities

Two-way sync of same data, as well as one-way sync

Conflict Handling

Detect and resolve conflicts caused by the same data being changed in multiple locations

Logging and Monitoring

Administration capabilities for tracking usage

Scale

Service scales as resources requirements grow

Currently there are 2 CTPs available

CTP 1 publicly available
CTP 2 invitation only
CTP 3 will be available by the end of the summer and will be feature complete

Geo Available Applications

Windows Azure Traffic Manager

This will help handle geo-locating SQL Azure Databases
This will manage the traffic

SQL Azure Data Sync

Syncs the SQL Azure Databases between regions

CY11 Investment Themes

Managed Services

Data Protection / Recovery

Local Backup/Point in Time Restore: to recover from user/apps errors
Geo Backup/Disaster Recovery from data center/geographical disasters

New Data Movement Capabilities

DB Import/Export of data and schema between SQL Azure and SQL Server
Migration Support to SQL Azure from Sybase, in addition to existing support for Access, MySQL and Oracle

Enhanced Management Experiences

Enhanced portal based experiences
Multiple DB administrators per subscriptions to support enterprise scenarios

Scale on Demand

Scale-out with Federations

Scale to virtually unlimited size to host very large databases across multiple servers
Better support for multi-tenant applications with numerous tenants

Scale-up and Scale-down Databases

Scale beyond 50GB and less than 1GB

Performance Predictability

Reserve compute capacity with Resource SLA

Enhanced multi-tenancy support

API level support for multiple DB server per subscription, provisioning & de-provisioning

Faster Innovation

Rich Insights with Reporting as a Service

Ability to author rich reports, host in Azure and view in on-premise or cloud apps

Spanning Apps with Sync Service

Synchronize data between SQL Azure DBs between SQL Azure & SQL Server

Rich Developer Experiences

Professional Devs: VS “Juneau”
Department Devs: LightSwitch
Java Devs: supported JDBC driver
International Devs: Localized Azure Portal

Data Market Enhancements

International rollout
Enhanced Content Management
Integration with Denali, Office and CRM

Resources

http://www.sqlazure.com
http://blogs.msdn.com/sqlazure
http://www.sqlazure.com/community – Tutorials
http://www.sqlazure.com/getstarted – Free Trial

Windows Azure Security Essentials – Part 1/N – Security Overview

NunoGodinho — Tue, 08 Feb 2011 23:35:27 GMT

Security is one of the very important parts of Cloud Computing, since we are in a completely new environment that the ones we are used to. When using Cloud Computing “The Platform is managed for us” rather, the “Us managing our platform”.

Another thing that is completely different between the On-Premise and the Platform as a Service Model, is because instead of having full control of the platform (Physical/Network/Host/Application and Data), we are really only controlling the Data and the Application, since the rest is the responsibility of our vendor, in the Windows Azure Service Platform case this is Microsoft. But having this managed by Microsoft, we need assurance that our data is secure, and protected even from Microsoft Admins.

So some of the Cloud Security Concerns are:

Where is my data located? Is the Data Stored in some place that is possible based on my country laws and regulations?
Is the Microsoft Cloud “secure”?
Who can see my data?
How do you make sure my company data follows “the rules”? (Country rules and laws)

And for all of this Cloud Security needs a Mind Shift, since:

Much of the traditional infrastructure security moves to the platform and application layers, and we are unable to change them. So we need to do something different like:

So Network Access Control Lists and Firewalls become host packet filters and virtual filters
Reduction of the attack surface, least privilege, user authentication and input sanitization become key concerns when designing and developing an application.
Also the Platform and Network Level encryption will still play a very important role, but the most important point is that the developer becomes more responsible for the application security and encryption design.

In order to manage the Security on Windows Azure we can use:

Service Management Security

Customers create a Windows Azure subscription using LiveID credentials

Live ID is one of the longest running Internet Authentication Services available

Hosted services and storage accounts managed through LiveID or a Service Management API or SMAPI with user-generated public/private key pairs.

SMAPI protocol runs over SSL and it’s authenticated based on the user-generated public/private keys.

All communications between the several Windows Azure internal components are encrypted using SSL, and also the communication between the Fabric Controller and the managed nodes is unidirectional.

By default the Web Roles run in partial trust mode, and the recommendation is that all public facing roles should use Partial Trust as well as avoiding to have secret information. Instead we should really pass that information to a higher level trust role, for example to a Worker Role running on Full-Trusted mode and only with Internal Endpoints.

Within each Windows Azure Subscription a developer can create multiple storage account, and each account has 2 Keys in order to provide the key rollover capability, the same way as we change our passwords frequently in order to maintain security.

SQL Azure Security

Same security Model as in SQL Server Authentication Model

SQL Server Logins

Authenticate access to SQL Azure

Database Users

Grant access to the database level

Database Roles

Group users and grant to the database level

Only SQL Server authentication is supported

User must provide credentials every time they connect
Password resets will not force a connection to be re-authenticated
Every 60 minutes has passed since last authentication, a re-authentication is performed (done automatically, and if there is a password update, this will be used instead)

During the provisioning process, SQL Server created a login for you that is the server-level principal similar to SA (System Administrator) login in SQL Azure.

Used to create additional user accounts

Only available thought port TCP 1433, which needs that customers configure their internal firewalls to allow Outgoing request to this port.
Source IP address needs to be authorized in SQL Azure Firewall

PDC10 – Keynote

NunoGodinho — Thu, 28 Oct 2010 18:29:46 GMT

A New Era of Opportunity

Steve Ballmer – CEO @ Microsoft

Device: Windows 7, Windows Phone 7, XBOX 360

Internet Explorer and HTML5 is the glue to bring all of them together

Cloud: Windows Azure, SQL Azure, XBOX Live, Bing, Office 365, Windows Live

Internet Explorer 9 and HTML5

Features
- Rich Graphics
- Next Level of Interactivity
- Same Markup
Fast, Clean, Trusted and Interoperable
The most downloaded Beta ever

Dean Hachamovitch – IE 9

Samples:
- KFXP
- Track Splash
Pin Sites to the Taskbar
Interact with Windows 7 Jump Lists, Sneak Peek, Previews and so on using JavaScript
IE9 Platform Preview 6 Available Today – http://ietestdrive.com

Windows Phone 7

Different kind of phone, UX paradigm
New developer perspective
User as the Central point of the UX

Amazon Kindle App for Windows Phone 7

Scott Guthrie – VP @ Microsoft – Developing Windows Phone 7 App

New Capabilities in the Future of Windows Phone 7 Tools
- Performance Analysis Tool
  - Available information out-of-the-box:
    - Frame rate
    - CPU Usage
    - Storyboard

Windows Azure with Bob Muglia – President @ Microsoft

Cloud Computing this really gives us IT as a Service
- IaaS
- PaaS
- SaaS
Comparing Platforms Today and PaaS

Focus/Model Platform Today Platform as a Service (PaaS)

Management Server, VM, Network, Storage, App App

Maintenance Patch, Service Release, New Version Maintained for you

Deployment Assembly Required Read-Made Services

Deployment Custom, Inconsistent Standardization

Performance Plan for Peak Load On-Demand Scale

Failure Built to avoid and recover from failure Built to

Windows Azure
- Foundation: Compute, Storage, Networking, Identity & Security
- Services: Web, HPC, Caching, Messsaging, Workflow, Content Delivery
- Information: Relational Databases, Reporting & Analysis, Data Sync
- Tools: Visual Studio, System Center
Windows Azure Platform
- Windows Azure
- SQL Azure
  - SQL Server: Designed to run on one server or in a cluster of Servers
  - SQL Azure: Designed to run on thousands of servers across Data Centers with backups and so recoverability and so on
- .NET
- Visual Studio
- System Center
You Decide: Language, Development Framework, Development Tool, Management Tool, Data Center (Windows Azure Appliance)
Vision: It’s All about the App
- Move
  - Announcing:
    - Windows Azure Virtual Machine Role
    - Server Application Virtualization
    - Image Creation in the Cloud coming up next Year
- Enhance
  - Announcing:
    - Extra small Instance
    - Remote Desktop
    - Full IIS
    - Startup Tasks
    - Virtual Network
    - Elevated Privileges
    - Windows Server 2008 R2 Roles
    - Multiple Administrators
    - Windows Connect
      - Access to on-premise resources for cross-premise apps
      - Support for Domain-joining VMs
      - Direct role-instance connectivity for easier development
      - Use your existing remote administrator tools
    - Windows Azure AppFabric
      - Access Control (Support for OAuth, LiveId, GoogleId, Facebook, …)
      - Caching
        
        Windows Server AppFabric Caching in the Cloud
        
        Same API as the On-premise version
      - Service Bus ( now supports Biztalk connections )
    - SQL Azure
      - Reporting Services
        
        Standard Report, and everything is the same
        
        Publish you Reports into the Cloud
      - Data Sync
      - What’s new in SQL Azure
        
        pre-Recorded Video - http://player.microsoftpdc.com/Session/91e8c18c-b223-4695-a70d-eaa762766db9
      - What’s new in SQL Azure Reporting Services
        
        pre-recorded Video - http://player.microsoftpdc.com/Session/5007e9c3-03cd-41b4-9e1c-4eb17cd60e37
    - New Windows Azure Platform Management Portal
    - Multiple users & roles for management
- Transform
  - Windows Azure AppFabric
    - Announcing:
      - Composite App Service & Workflow
        
        You can use this to composite apps using services not only from Azure
      - Composition Model
      - Tools
        
        Visual Tools for Visual Studio 2010 for modeling your
Windows Azure Connect plugin is very cool
Visual Studio Team Foundation Server on Windows Azure
- CTP available Next year
- http://blackstock.cloudapp.net/
Announcing Windows Azure Marketplace TODAY
- Data Market (Previously known as Project “Dallas”
  - 35+ Content partners
  - 60+ Data Offerings
  - 100+ Offers
  - Supported via OData
  - http://datamarket.azure.com

Check SQL Azure Database Associated Cost

NunoGodinho — Tue, 19 Oct 2010 00:16:42 GMT

Ok so I found out exactly the size of every table in my SQL Azure Database, but now I’d like to know a little bit more, since I’m paying and I want to know how the payment is related to each of my tables. So in order to do this and after reading this blog post from the SQL Azure Team, this is the statement I’ve come up with:

DECLARE @SizeInBytes bigint
SELECT @SizeInBytes =
(SUM(reserved_page_count) * 8192)
FROM sys.dm_db_partition_stats

DECLARE @Edition sql_variant
SELECT @Edition =DATABASEPROPERTYEX ( DB_Name() , 'Edition' )

Select @Edition, @SizeInBytes

So now I have the Size used in Bytes and the Edition of the SQL Azure Database that I’m using, and so I can now calculate how much is the cost per byte.

DECLARE @CostPerByte float
SELECT   @CostPerByte = (CASE
    WHEN @SizeInBytes/1073741824.0 < 1 THEN (CASE @Edition WHEN 'Web' THEN 9.99 ELSE 99.99 END)
    WHEN @SizeInBytes/1073741824.0 < 5 THEN (CASE @Edition WHEN 'Web' THEN 49.95 ELSE 99.99 END)
    WHEN @SizeInBytes/1073741824.0 < 10 THEN 99.99
    WHEN @SizeInBytes/1073741824.0 < 20 THEN 199.98
    WHEN @SizeInBytes/1073741824.0 < 30 THEN 299.97
    WHEN @SizeInBytes/1073741824.0 < 40 THEN 399.96
    WHEN @SizeInBytes/1073741824.0 < 50 THEN 499.95
         END) / @SizeInBytes

Ok. Now I know exactly what is the cost per byte, and so what I need to know is exactly how much cost is associated with each Table.

select sys.objects.name, @CostPerByte * (sum(reserved_page_count) * 8192)
from sys.dm_db_partition_stats,
sys.objects
where sys.dm_db_partition_stats.object_id = sys.objects.object_id
group by sys.objects.name

So with this I know exactly how my cost is being split by each table. So now I only want to see if everything in the end comes up to the correct result.

Select ROUND(SUM(Cost.TotalCost), 2)
from (    select sys.objects.name TableName, @CostPerByte * sum(reserved_page_count) * 8192 TotalCost
        from sys.dm_db_partition_stats,
             sys.objects
        where sys.dm_db_partition_stats.object_id = sys.objects.object_id
        group by sys.objects.name) Cost

Ok. So the result is correct. Sweet.

Check SQL Azure Database Size

NunoGodinho — Mon, 18 Oct 2010 23:26:39 GMT

I was looking at my SQL Azure Database and suddenly I saw that I had to clean some space because it was already beyond the maximum quota that I had defined for the Database, but I needed to know exactly where the most space was being used in order to better understand not only what was the size, but why and where was the space being used.

So my first action was, Ok so I’ll use the storeprocedure sp_spaceused.

Exec sp_spaceused

So with this I know exactly the space that my database is using, and if I use this:

Exec sp_spaceused <TableName>

I’ll get the space that is being used by the Table that I specify.

Ok. So I tried this on my local SQL Server and all was good, and so off to SQL Azure, and I get the following error:

Could not find stored procedure 'sp_spaceused'.

So. Not good. How will I get the needed results? So I used the following statement:

select sum(reserved_page_count) * 8.0 / 1024
from sys.dm_db_partition_stats

And this statement gives me the exact size that I have in my SQLAzure Database, but I need to know exactly how much space I have in each table, so I use the following statement:

select sys.objects.name, sum(reserved_page_count) * 8.0 / 1024
from sys.dm_db_partition_stats,
sys.objects
where sys.dm_db_partition_stats.object_id = sys.objects.object_id
group by sys.objects.name

This last statement gives me a list of all tables that exist in the specified SQL Azure Database and the exact size that they have.

I used this based on Ryan Dunn’s blog post available here.

Hope this helps you.

PDC09 – Day Two – Developing Advanced Applications with Windows Azure

NunoGodinho — Wed, 18 Nov 2009 22:56:04 GMT

Steve Marx – Technology Strategist

Now we can have the ability to have more than a Web and a Work Role, since we have several different types of Roles.

Example:

IIS Role
SMTP Role
…

Now the there is the new Storage API that is currently being shipped with the SDK and not only a sample.

Some new features:

Inter-role communication
Non-Http endpoints
Role instance lifecycle
VM Sizes
Full-trust in the Cloud
Local Storage

Upgrading the local storage

How to model you application

Draw the boxes and arrows
Each box is a role

Which receive traffic from the internet
Which need IIS?

Receive Traffic: Input Endpoints

Different ports on same domain
There’s always load balancing
Declared in ServiceDefinition.csdef
Handled by IIS

InputEndpoint

Defined the protocol, port and name of the endpoint that will opened in order to listen for the requests
Protocols supported

Http
Https
Tcp

RoleEnvironment is the class that allow us to communicate with the Windows Azure fabric, for example in order to know the Configurations

Aside about Email

Spammers will use Windows Azure then the IP ranges can get blacklisted

Best Practices: Use a relay service in order not to have your mails marked as SPAM

Send mail via a rely mail
Receive mail is fine

Searching

Lucene.NET

Popular search API
.NET port of original Java implementation

Index data

Bring index up to speed on startup
Continually update index

Serve searched (via WCF)

Asynchronous Work without a Queue

Queues deliver messages to one worker only
We need every worker to see the message

Solution

use Table instead

Build your own secondary index
Order references

Initialization of Roles

Now we have:

OnStart() – “busy” state, initialization
Run() – do work
OnStop() – traffic stops, graceful shutdown

Internal Endpoints

Declare endpoints in ServiceDefinition.csdef
API to find out what port to listen on RoleEnvironment in Order to know if the port or anything change about the endpoint since the last call

How to Get More RAM

RAMDiectory pouts everything in memory
Choose the VM Size:

Small
Medium
Large
Extra-Large

PDC09 – Day One – SQL Azure Present & Futures

NunoGodinho — Tue, 17 Nov 2009 23:37:57 GMT

Data Platform as a Service

In the future will have Reporting, Data Analytics

SQL Azure Database

Relational Database service

SQL Server technology foundation
Scalable
Symmetrical

SQL Azure Provisioning

Each account has zero or more servers

Azure wide provisioned in a common portal
Billing instrument

Each server has one or more databases

contains metadata about the databases
Unit of authentication and security
Unit of Geo-location
The server is a logical grouping of you DBs

Each database has standard SQL Object

Unit of consistency
Contains users, tables, views, indexes, and so on,

Global Availability

North Europe
Southeast Asia
North and South USA

Futures

Goal: Provide and end-to-end experience for enterprise departmental apps and Saas ISVs scenarios
Improved Tools

Today we have

VS, SSMS, ADO.NET, ADO.NET Data Service

Data Backups

Multiple replicas of all databases with automatic failover. System backups for additional protection
Use Database Clone to create a clone of the Database with every capabilities of the original one
During 1H 2010

Continuous Backups

Configurable retention and lag period
Backups available for read operations
Local and regional options
During 2H 2010

Operational Models

SaaS

Provisioning APIs for ISVs
Template database support
Meta-data tracking
Additional billing scenarios

Scale-out support –Today

Databases and workloads partitioning is a classic

Better price/performance
Levels of throughput not possible with a single machine

What SQL Azure addresses

Highly available service on top of commodity hardware
Zero administration

There will be available several options and the 10GBs of space is not an issue

Customer Feedback

Add support for profiler, DMV’s
SQLCLR
and much more

Codename “Vidalia”

Sharing to trustworthy collaboration

Cloud is a grate place to connect
Empowers multiple parties
Businesses need control

Very interesting demos, need to see more about it.

Focus/Model	Platform Today	Platform as a Service (PaaS)
Management	Server, VM, Network, Storage, App	App
Maintenance	Patch, Service Release, New Version	Maintained for you
Deployment	Assembly Required	Read-Made Services
Deployment	Custom, Inconsistent	Standardization
Performance	Plan for Peak Load	On-Demand Scale
Failure	Built to avoid and recover from failure	Built to