Azure Service Bus is an asynchronous cloud messaging service that connects Data stores, Service Apps, and Devices across public and private clouds.
Let’s try to understand why we need an architecture such as a Service Bus.
Simple Stove-piped Applications
Often, we build applications that are specific for a purpose and are self-contained. Such applications have 3 or more-tiers for UI components, Business logic and an Application Database. Since the database is intrinsic to the application and is usually locally hosted, data access is easy and fast. This is the simplest form of architecture but probably not the best for growing businesses.
Figure 1 Stove-piped Application Architecture
Consider there is another stove-piped application with its own database. This database shares some tables with the first database, in the sense, ‘Contact’ table in Application A essentially saves the same data as ‘Customer’ table in Application B. The reason they are named differently could be because the applications were built by different teams at different times, this happens all the time. Both applications are now keeping copies of same data ‘Contact/Customer’ in two different disconnected places. There is data redundancy, also the names and schemas of the tables are different. There is also business rules redundancy, as the data that is supposed to be same should theoretically call same structural business rules. Synchronizing data between such autonomous applications can become almost impossible without custom integrations.
Figure 2 Stove Piped applications maintaining data synchronizing using custom Integrations
Custom integrations are never an easy task. In a 2-way integration, one needs to account for any data changes in application and carry them to other application. And vice versa. There could be simultaneous updates and one needs to handle those with first come, first win basis or by priority preferences. In big enterprises, there are more than 2 applications that need data synchronization and integration complexity increases multifold. Moreover, Integration Solutions do not solve the problem of Business Rules redundancy, that is still a problem.
Figure 3 Multiple Stove Piped applications using Custom Integration for Data Integrity
Is there a way to overcome the need for custom, heavy and expensive integrations? There is, by having a Master Data Store (Microsoft’s Common Data Model is an effort in that direction). Having data stored in one place and asking applications to seek data from it (via a service called Information Service) would be helpful to overcome the integration hell. Applications may also keep a Read-Only copy of the data for query purposes. This read-only version may refresh data at an interval to maintain the data consistency or it could only consist of tables that don’t change that often. Application will directly go to Master data source if data changes. This model also allows the structural business rules to reside with the Master database. Since these rules need to be operated on the shared data, they can be shared by seeking applications as well.
Figure 4 Applications using a Master copy of Data via Information Services
Now that Applications are not autonomous anymore, there is some data latency as compared to data being locally available. Also, if there are more than one Information Services providing data to the application then applications need to maintain clusters locally to join data from different services. We may introduce an additional layer that would do the heavy lifting and collect data from different services and structure the data as expected by the applications. But this way we are just passing the complexity to another layer and not reducing or removing it.
Is there a way where we can have the best of both worlds, avoiding the worst? A Master data storage to remove the need for complex data integration and yet have all the data an application needs in one place for faster data access? Service Bus comes for the rescue!
Just before we jump to asynchronous pattern of Service Bus, let’s look at synchronous Publisher-Subscriber model of propagating data changes. So far, we have applications that uses the endpoints exposed by Information Services to update data. Applications use local copy of read-only database for querying purposes. How can this read-only be maintained to have latest data? Publisher Service sitting in front of Information Services would take care of sending notifications to all the subscribers at their end points.
Figure 5 Information Services providing updates to read-only database copies in multiple Applications using a Pub-Sub model
Disadvantages to this model is that Publisher needs to know all the subscribers before-hand, also what happens if a Subscriber is down or removed, or a new subscriber is added? Publisher needs to be re-configured to handle these situations. There is a need for maintenance. Since this is a synchronous model, Publisher must send notifications one at a time, which adds time and wastes resources. If there is an additional publisher available then Application needs to develop an API to receive subscription updates from the new Publisher.
Azure Service Bus
Service Azure Bus provides two features Messaging and Push Notifications (Broadcasting). Messaging has 2 kinds, Relay Messaging and Brokered Messaging. Relay messaging allows communication between two or more web services even through firewall. In this blog, we are going to discuss Brokered Messaging in detail.
Service Broker Queues and Topics provide decoupled messaging features that support Pub-Sub (publish-subscribe) model. In this pattern there is a Service Broker which get a message from a sender and transport it to the receiver using Queues. Sender need not know who the receiver is, it simply out a message on the Sender Queue; Receiver does not see the sender, it picks messages from Receiver Queue.
Azure Service Queues, Topics and Subscriptions
A Queue follows First in – First out paradigm. When a message is pushed on the Queue, it gets queued behind other existing messages, and these messages are picked by Receiver, one at a time. Key benefit of Queues is that sender does not have to send messages at the same time receiver is receiving them. Queues provide a storage space for messages. Sender and Receiver can perform actions at different pace and frequency. Also, Sender does not have to wait for an acknowledgment from a Receiver.
Topics are Queues too but with a variation. Each published message is copied to a separate virtual Queue built for each Subscriber. Topics provide one too many forms of communication, similar to Pub-Sub model. Each subscriber can specify its own filtering conditions.
Figure 6 Azure Service Bus Topic
Azure Service Bus and Dynamics 365
Azure Service Bus provides a native integration with Dynamics 365. This means we can send messages to Azure Service Bus from CRM when an event occurs, these messages can then be used to integrate with several downstream applications.
SAS (Shared Access Signature) authentication is used to access Service Bus resources. SAS authentication involves configuring a cryptographic key with associated rights on a Service Bus resource. Clients such as Dynamics CRM can get access to that resource by presenting a SAS token.
Let’s build a CRM to Azure Service Bus integration.
Step 1: Configure Azure Service Bus and obtain a connection string
You will need an active Microsoft Azure account. Browse to the Azure Portal, and click Service Bus. We will start by creating a new namespace.
Next, create a new Service Bus instance.
Next, create a new Service Bus Messaging Entity such as Queue.
Add new SAS Policy
Finally, obtain the connection string by browsing to newly created Shared Access Policy.
Step 2: Use CRM plugin tool to make a connection to CRM and Azure Service Bus
To get the Registration tool, download Microsoft Dynamics 365 SDK. Look for the tool in folder Tools\PluginRegistration folder. Run the tool.
Create new connection and login to your Dynamics CRM 365. Next, register a new Service Endpoint.
Copy and paste the connection string obtained in Step 1.
New Service Endpoint now appears in the list of registered plugins.
Step 3: Define events which will post messages to Azure Service Bus
Right-click the Service Endpoint and add a new step. Enter trigger action such as create, delete, etc. in the ‘Message’ box. Enter the primary Entity on which the action will occur. Also, mark the execution mode as ‘Asynchronous’.
Once the new Service Endpoint is all configured, now it is time to test it out. Login to you Dynamics CRM 365 portal and go to Accounts. Create a new account. This should trigger an action on the Service Endpoint.
Verify that a message is queued on the Azure Service Bus. Go to Settings -> System Jobs and an event should be lined up there.
Azure Service Bus shows 1 message lined up in the Queue to be picked up by a subscriber.
We can now subscribe any downstream applications, Logic Apps, 3rd part legacy applications to subscribe on this Queue and pick up the message. If there are more than one subscriber then the message can be put on a Topic instead of a Queue.
And that’s how you can easily configure Azure Service Bus for Dynamics CRM.
Looking for more Azure integrations with Microsoft? Or want more help integrating Azure service bus for Dynamics?
We’d love to hear from you.
