November 2010 - Posts
Clemens Vasters – Principal Technical Lead @ Windows Azure Team
Service Bus Today
- Naming fabric or Structure that sits out on the internet that provides us to create a namespace
- When we create a namespace for the service bus, Microsoft provisions a tenant in a multi-tenant value fabric for us, where we can create endpoints, message buffers, and communicate through those.
- Around this routing fabric we have several nodes where we connect and that’s is where the fabric is being used.
- Use Case Samples:
- Message Queues: Wherever you need some application queue
- Airline that is using this to check-in and synchronize devices using this
- Outside Services: Whenever you need to provide a public facing service, but your internal service
- MultiCast (Eventing)
- Direct Connect (Bridge two environments through Firewalls and NAT.
A look into the Future
- Service Bus Capability Scope
- Connectivity (currently in the CTP)
- Service Relay
- Protocol Tunnel
- Eventing
- Push
- Messaging (currently in the CTP)
- Queuing
- Pub/Sub
- Reliable Transfer
- Service Management
- Naming
- Discovery
- Monitoring
- Orchestration (Workflow)
- Routing
- Coordination
- Transformation
- Service Bus October 2010 Labs (labs release not parity with production services)
- New and Improved
- Load Balancing
- I need to be able to take multiple listeners and host them on Service Bus and have load balancing on them
- This is not possible on the production environment, and in the Labs we have a new protocol called AnyCast to provide this.
- Richer Management
- Durable Message Buffers
- We want to be more queues and less about message buffers
- In this October 2010 Labs Release we don’t have:
- HttpListeners
- Capabilities in the Service Registry
- Missing some composition capabilities
- Missing all the bindings
- The purpose is to gather feedback about this new protocol
| Version | Storage | TTL | Capacity | Messages |
| Production | In-Memory | 10 mins | 2 MB | 60KB |
| Labs | Durable and Replicated Storage | No Limit | 100 MB | 256KB |
- Same Lightweight REST protocol
- Long pooling support
- Future Releases
- Reliable Transfer protocol options
- Higher throughput transport options
- Volatile buffers for higher throughput
- Currently the same protocol backed up with a REAL Queuing system that is reliable and capacity.
- Listener Load Balancing
- Connection point management separate from listener
- Explicit Management of service bus connection points to the service bus
- We gain the ability to know if someone is listening or not.
- Multiple listeners can share the same connection point
- Load balancing and no single point of failure
- Sticky sessions
- Session Multiplexing
- One socket per listener
- Optimized for short sessions, and short messages
- The latency for setting up that session is now very short
- No extra round-trip to set up a session
- Note: The purpose is to reduce latency
- Explicit Streaming Support (Default model in production Today)
- One socket per client
- Optimized for long sessions, streaming and very large messages
- Lightweight handshake, mostly ‘naked’ end-to-end socket afterwards
- Note: This will be made available later into the Labs
- Namespace and Management
- Management Surface Today
- Refactoring Goals
- Management consistently explicit
- No more implicit connection points, just explicit
- Split management and runtime surface
- Enable richer addressing/organization
- In the Labs
- Protocols
- Currently allowed
- Currently thought
- FTP
- SMTP
- SMS
- Other
- Candidate Example of those: XMPP, … (not planned but thought as possible candidates)
- Messaging
- Reliable, transacted, end-to-end message, transfer of message sequences
- Local Transactions
- Batches are transferred completely or not at all
- Service Bus Pub/Sub – Topics
- Service Bus Topics
- Think of as a message log with multiple subscriptions
- Durable message sequences for multiple readers with per-subscription cursors
- Pull Model consumption using messaging protocols
- Service Bus manages subscription state
- Service Bus Pub/Sub – Eventing
- Service Bus Events
- We already have it with neteventrelaybinding
- Multicast event distribution
- Push delivery via outbound push or connectivity listeners
- Same subscription and filter model as topics
- Considering UDP for (potentially lossy) ultra-low latency delivery
- Service Bus Push Notifications
- Distribution points allow for push delivery to a variety of targets
- Built on top of pub/sub topic infrastructure
- Protocol adapters for HTTP, NET/TCP, SMS, SMTP, …
- Subscription-level AuthN and reference headers
- Monitoring
- Monitor of Service Bus state
- System events exposed as topics with pull and push delivery
- Events can flow into store for analytics
- Ad-Hoc queries into existing resources and their state
- Discovery
- Discovery of Service Bus resources and resource instances
- Eventing and query mechanisms similar to monitoring
- Service Presence
- Custom Taxonomies
What can you do know. Use AppFabric Service Bus Labs
Nina (Ling) Hu – Program Manager @ Sync Framework Team
Key Takeaways:
- Offline applications have lots of benefits comparing to online applications especially for mobile devices and cloud services
- Sync Framework, SQL Azure and Windows Azure provide a data platform that makes it very easy to build offline applications and sync services
- Sync Framework caters for offline on any client platform, and provides better end-to-end support for Windows, Silverlight and Windows Phone 7 clients.
Why to Build Offline capable Apps?
- Offline/Cached mode apps enjoy lots of value
- Client apps get better UX through
- Lower latency – data access doesn’t require round-trips
- Higher availability – App still runs if server is unreachable
- Network utilization is reduced – most data access is local
- Servers gain better ability to schedule work asynchronously
- Benefits of cached mode are magnifies on the internet because
- Server/service is further away
- Network is less tuned/reliable
- Lowers the need of scaling you app
Overview for Sync Framework v4
- Capabilities where extended to other platforms like:
- Windows
- SQL Server Express
- SQL Compact
- Silverlight
- Isolated Storage
- Other Storage
- Windows Phone 7
- Isolated Storage
- Other Storage
- Windows Mobile
- Browser 7 / HTML5
- iPhone / Any Client
- How was this achieved?
- Using a single protocol for doing the communication and Synchronization. ODATA
- What platforms have Client API support?
- Windows
- Silverlight
- Windows Phone 7
- All other platforms have minimal client & store requirements
- Basically there are made available a set of samples to show how to implement those.
- Over time there will be provided some tools to generate the code needed to integrate with you platform
Offline Applications Architecture
- On the Server
- ODATA Sync Framework
- Sync Logic
- SQL Azure Provider
- Business Logic
- This is the only part needed to be written for our application
- SQL Azure
- Steps to Build a Sync Service
- Provision the Database to use Sync Framework
- Launch the SyncSVCUtilHelper.exe
- Select the configuration file for the Synchronization
- Select the Name of the configuration file
- Provide the name of the database
- Create a Sync Scope
- Define if we enable filtering or not
- Select the Tables/Columns/Rows that you want to define the scope
- Provision the Database
- Select the configuration file
- Next
- Code Generation
- Select the Configuration file
- Specify the Side that you want the file to be Generated (Server/Client)
- Specify the Output directory
- Select the Language (CS/VB)
- Create the Sync Service
- Open VS
- Create a new ASP.NET Project
- Add Reference to Microsoft.Syncronization.Service.dll
- Add the Code generated by the Tool?
- Note: When you open the Service in the Browser if you add the $diag you’ll get a page with the diagnostics of the service, and see if everything needed is in place
- Overview
- Data Store
- Expose data from SQL Azure or SQL Azure
- Host
- Hosted on Windows Azure or IIS/Windows Server
- Exposed using WCF Sync endpoint
- Protocol
- Expose data for synchronization via a protocol and allows third-parties to build offline clients
- Features
- Support business logic
- Custom authentication / authorization
- Filtering
- Tooling
- Provide a Tooling wizard experience to configuring server and client
- On the Client
- Silverlight Offline Application
- Only this is needed to be developed
- Collections
- Isolated Storage
- Cache Controller
- OData Sync Proxy
- Isolated Storage Provider
- Support for Silverlight 3 and 4
- Steps to build a Sync Framework Client
- Create a new Silverlight Application
- Add Reference Microsoft.Synchronization.ClientService.dll
- Add Reference System.ComponentModel.DataAnotation.dll
- Add Reference System.Windows.Controls.Data.dll
- Open the Tool SVCSyncUtilHelper.exe
- Choose the previous Configuration file
- Select the Side you want to generate (Isolated Storage client)
- Select the Language (CS/VB)
- Add the Files generated to your application
- On the Client for other Platforms
- Track changes on the local storage (samples for common stores)
- Implement the client-side sync proxy (samples for sync proxies)
- Protocol
- OData Protocol & Sync
- Apply the principals of OData to the problem of data-sync
- Standardize on protocol not components
- Minimal client sync logic and algorithms
- Service manages sync keeping client simple
- Provide samples to how to consume protocol for any platform
- Provide components for richer experience in Silverlight & Windows Phone 7
- Full interop details defined in the SDK
Roadmap
- Sync Framework 4.0 CTP is available now.
- SQL Azure Data Sync CTP 1 its for the Cloud to Cloud Service
- SQL Azure Data Sync CTP 2 will support On-premise to Cloud Sync
- Sync Framework 4.0 RTW will be available on 2011
More resources
Anthony Chavez – Director @ Windows Azure Networking Group
Introducing Windows Azure Connect
- Secure network connectivity between on-premises and cloud
- Supports standard IP protocols (TCP, UDP)
- Example use cases:
- Enterprise app migrated to Windows Azure that requires access to on-premise SQL Server
- Windows Azure app domain-joined to corporate Active Directory
- Remote administration and troubleshooting of Windows Azure roles
- Simple setup and management
Roadmap
- CTP release by end of 2010
- Support connect from Azure to non-Azure resources
- Supports Windows Server 2008 R2, Windows Server 2008, Windows 7, Windows Vista SP1, and up
- Future releases
- Enable connectivity using existing on-premises VPN devices
Closer Look
- Three steps to setup Windows Azure connect
- Enable Windows Azure (WA) roles for External connectivity via service Model
- Select only the roles that should be enabled for external onnectivity
- Enable local computers for connectivity by installing WA Connect Agent
- Configure/Manage your network policy that defines which Azure roles and which Azure computers can communicate.
- defined using the Connect Portal
- After the Configuration/Management of the Network Polity, Azure Connect automatically setups secure IP-level network between connected role instances and local computers
- Tunnel firewall/NAT/s through hosted relay service
- Secured via end-to-end IPSec
- DNS name resolution
Windows Azure Service Deployment
- To use Connect for a Windows Azure Service, enable one or more of its Roles
- For Web & Worker Roles, include the connect plug-in as part of the Service Model (using .csdef file)
- For VM Roles, install the connect agent in VHD image using Connect VM Install package
- Connect agent will automatically be deployed for each new role instance that starts up
- Connect agent configuration is managed through the ServiceConfiguration (.cscfg file)
- One one configuration setting is required
- ActivationToken
- Unique per-subscription token, accessed from Admin UI
- Several Optional settings for managing AD domain-join and service availability
Deployment
On-Premise
- Local computers are enabled for connectivity by installing & activating the Connect Agent. It can be retrieved from:
- Web-based installation link
- Retrieved from the Admin Portal
- Contains per-subscription activation token embedded in the url
- Standalone install package
- Retrieved from the Admin Portal
- Enabled installation using existing software distribution tools
- Connect agent tray icon & client UI, enables us to:
- View activation state & connectivity status
- Refresh network policy
- Connect agent automatically manages network connectivity, by:
- Setting up a virtual network adapter
- “Auto-connecting” to Connect relay service as needed
- Configuring IPSec policy based on network policy
- Enabling DNS name resolution
- Automatically syncing latest network policies
Management of Network Policy
- Connect network policy managed through Windows Azure admin portal
- Managed on a per-subscription basis
- Local Computers are organized into groups
- Eg. “SQL Server Group”, “Laptops Group”, …
- A computer can only belong to a single group at a time
- Newly activated computers aren’t assigned to any group
- Windows Azure roles can be connected to groups
- Enabled network connectivity between all Role instances (VMs) and local computer in the Group
- Windows Azure connect doesn’t connect to other Windows Azure Roles
- Groups can be connected to other Groups
- Enabled network connectivity between computers in each group
- A group can be ‘interconnected’ – enables connectivity within the group
- Useful for ad-hoc & roaming scenarios
- Eg. your laptop having a secure connection back to a server that resides inside the corp net
Network Behavior
- Connect resources (Windows Azure role instances and external machines) have secure IP-level network connectivity
- Regardless of physical network topology (Firewall / NATs) as long as they support outbound HTTPs access to Connect Relay service
- Each connected machine has a routable IPv6 address
- Connect agent sets up the virtual network address
- No changes to existing networks
- Communication between resources is secured via end-to-end certificate-based IPSec
- Scoped to Connect Virtual network
- Automated management of IPSec certificates
- DNS name resolution for connected resources based on machine names
- Both directions are supported (Windows Azure to Local Computer or vice-versa)
Active Directory Domain Join
- Connect plug-in support domain-join of Windows Azure roles to on-premise Active Directory
- Eg. Scenarios:
- Log into Windows Azure using Domain Accounts
- Connect to on-premise SQL Server using Windows Integrated Authentication
- Migrate LOB apps to cloud that assume domain-join environment
- Process:
- Install Connect agent on DC/DNS servers
- Recommendation: create a dedicated site in the case of multiple DC environment
- Configure Connect plug-in to automatically join Windows Azure role instances to Active Directory
- Specify credentials used for domain-join operation
- Specify the target OU for Windows Azure roles
- Specify the list of domain users / groups to add to the local administrators group
- Configure the network policy to enable connectivity between Windows Azure roles and DC/DNS Servers
- Note: New Windows Azure role instances will automatically be domain-joined
Finally the recap of Windows Azure Connect
- Enables secure network connectivity between Windows Azure and on-premise resources
- Simple to Setup & Manage
- Enabled Windows Azure Roles using connect plug-in
- Install Connect agent on local computers
- Configure network policy
- Useful Scenarios:
- Remote administration & troubleshooting
- Windows Azure Apps Access to on-premise Servers
- Domain-join Windows Azure roles
Adam Wilson – Program Manager @ Windows Azure DataMarket Team
Showing off different and Interesting integration scenarios with DataMarket
- Facebook application that integrates with a Data.Gov DataSets
- Usage of Windows Azure DataMarket Portal to explore the DataSets and also to show off the Tableau application to consume the DataSet from DataMarket
Integration Scenarios
- Choices:
- Specific dataset versus generic
- Influences the choice over the type of client technology to be used
- WebRequest – need to parse the XML by hand
- WCF Data Services client – ODATA client
- DataMarket service proxy classes
- Which DataMarket Account?
Building a simple Windows Phone 7 application
- App that uses two DataMarket DataSets, one about weather forecast and another about the population density
- Choose the DataMarket DataSet
- Explore the DataSet
- Test the DataMarket DataSet
- Choose the Type of Integration to use
- In this case the WebRequest is the one that is chosen
- Copy the Query URL
- Pass the Account Id and Account Key to the WebRequest
- Perform the Request
- Analyze the Response and Extract the XML elements that you need from the response.
- Each content provider deliver a set of help resources like samples, and documentation about the respective DataSet that are accessible on the DataSets Page
- Notes:
- Can be built using WebRequest directly
- Consider service-side
- Filtering – Filter only what you need and cache your results on the server to lower the bandwidth
- Projections – Do Projections on the Server-side in order to send only the elements that are interesting and needed by the client, reducing the used bandwidth
Next Steps: