Cloud Microsoft

Azure Messaging and Event Services

Publish Date: August 2, 2019

Microsoft Azure offers four services for messaging and events in Azure: Service Bus, Storage Queues, Event Hubs, and Event Grid.
Of these, two are specifically for events – Event Hubs and Event Grid, and the other two — Service Bus and Storage queues are for Azure messaging. Before discussing these Azure Services – let us describe Events and Messages.

Messages	Events
A message is raw data produced by a service to be consumed or stored elsewhere.	An event is a lightweight notification of a condition or a state change.
The publisher of the message expects how the consumer handles the message.	The publisher of the event does not expect how the event is handled
For example, A file sent as a message containing the data.	For example, An event notifies consumers that a file gets created. It may contain general information about the file but might not have the file itself.

Which service to use and When?

Service	Purpose	Type	When to use
Event Grid	Reactive programming	Event distribution (Discrete)	React to status changes
Event Hubs	Big data pipeline	Event streaming (Series)	Telemetry and distributed data streaming
Service Bus	HHigh-value enterprise messaging	Message	Order processing and financial transactions
Storage queue	Standard queuing scenarios, load leveling, and building process workflows	Message	Applications which need to store large sizes of messages in a queue.

Azure Event Grid
Azure Event Grid is used when it is required to build applications with event-based architectures. You can select Azure services for which you want to receive events and then select the endpoint to send them.
Built-in support is provided by Event Grid for events from Azure services, like storage blobs and resource groups. Event grid also has support for your events in your application using custom topics.
Event grid also provides filters for the event so that you can filter the required events for processing. Filters can also be used to route particular events to different end-points, multicast to multiple endpoints, and ensure your events are reliably delivered.
Five concepts in Azure Event Grid that let you get going :

Events– What happened?
Event sources– Where did the event took place.
Topics– The endpoint where publishers send events.
Event subscriptions– The endpoint or built-in mechanism to route events, sometimes to multiple handlers. Subscriptions are also used by handlers to filter incoming events intelligently.
Event handlers– The app or service is reacting to the event.

Where to use Event Grid
Serverless application architectures using Event Grid
Data sources and event handlers get connected by the Event. For example, use Event Grid to instantly trigger a serverless function that analyzes images when added to a blob storage container.
Ops Automation using Event Grid
Automation process can be speeded using Event Grid. For example, use Event Grid to notify Azure Automation when a virtual machine or SQL database gets created. Use the events to check/verify if service configurations are compliant automatically. Put metadata into operations tools, tag virtual machines, or file work items.
Application integration using Event Grid
The application can get connected with other services using Event Grid. For example, create a custom topic to send your app’s event data to Event Grid, and take advantage of its reliable delivery, advanced routing, and direct integration with Azure.
Use Case:
There is a Content Management System application that uploads thousands of media files every day. Admin wants to get notified whenever a new file gets added/modified in Azure storage via email. We can create a Logic App that monitors the changes to storage account and sends emails about these changes.
When you create a Logic App with an event subscription for an Azure resource, events flow from that resource through an event grid to the Logic App. We can also apply various filters for the events. E.g., Send the events only when a file gets added or modified.

Azure Event Hubs
Azure Event Hubs is a hyper-scale telemetry ingestion service that collects, transforms, and stores millions of events. As a distributed streaming platform, it gives you low latency and configurable time retention, which enables you to ingress massive amounts of telemetry into the cloud and read the data from multiple applications using publish-subscribe semantics.
It can receive and process millions of events per second. Data sent to an Event Hub can be transformed and stored by using any real-time analytics provider or batching/storage adapters.
Why Event Hubs
Event Hubs is perfect for analyzing big data. The amount of data can be vast and impossible to crunch, without a managed event service that has elastic scale capabilities. By the term “elastic,” we mean that it can handle spikes when they occur and it works with load profiles of different sizes. Apart from this, the key features are:

Fully managed PaaS
Support for real-time and batch processing
Scalable

Key architecture components:
Event Hubs contains the following key components:

Event producers: — Any entity that sends data to an Event Hub. Event publishers can publish events using HTTPS or AMQP 1.0 or Apache Kafka (1.0 and above).
Partitions: Each consumer only reads a specific subset or partition of the message stream.
Consumer groups: A view (state, position, or offset) of an entire Event Hub. Consumer groups enable consuming applications to have a separate view of the event stream. They read the stream independently at their own pace and with their offsets.
Throughput units: Pre-purchased units that control the throughput capacity of Event Hubs.
Event receivers: Any entity that reads event data from Event Hubs. All Event Hubs consumers connect via the AMQP 1.0 session. The Event Hubs service delivers events throughout a session as and when they become available. All Kafka consumers connect via the Kafka protocol 1.0 and later.

The following figure shows the Stream processing architecture of Event Hubs:

Event Hubs stream processing architecture

Use Case:
Several IoT devices send weather-related data, and it is required to store the streaming data and also stream the data with stream analytics so that it can get shown on the dashboard. The data can be ingested into Event Hubs and then from Event Hubs to storage and stream analytics.

Azure Service Bus
Azure Service Bus is a messaging service on the cloud, used to connect any applications, devices, and services in the cloud to other applications or services. It acts as a messaging backbone for applications in the cloud or across any devices.
Service Bus Concepts:
Some typical messaging scenarios are:

Messaging: Transfer business data, such as sales or purchase orders, journals, or inventory movements.
Decouple applications: Improve reliability and scalability of applications and services (Client and service do not have to be online at the same time).
Topics and subscriptions: Enable 1: N relationships between publishers and subscribers.
Message sessions: Implement workflows that require message ordering or message deferral.

The critical capabilities of the Service Bus are:

Queues: Queues offer an asynchronous messaging ability with first-in-first-out (FIFO) message delivery. Only one consumer receives one message.

Here a sender will send a message to a queue and the receiver will collect it in the initial or later stages from that queue. Besides this, queues enable us to store the message until the receiver is available to receive and process them. The user will receive the message upon request.

Topics and subscriptions:Topics and subscriptions offer a similar capability as queues. However, there could be more consumers for messages sent to a topic. A topic has one or more subscriptions, and they can have filters, and thus, various messages to a topic can end up in different subscriptions belonging to the topic.

Relay: In Service Bus Relay, we have an endpoint that is hosted in the cloud, but the difference would be the fact that our on-premise service will make an outbound connection to register with the Service Bus relay endpoint. At a conceptual level, we send a message to an endpoint in the cloud from where it will be forwarded to an on-premise service. As a result, it gives us the advantage to connect from one organization to another organization without having to set VPNs.

Service Bus Relay can be used to solve problems in scenarios like,

Information passed between two data centers.
Building integration architecture through SAP to forward messages into micro-service architecture using Service Bus Relay.

Advanced features:
Message sessions:
Sessions are used to realize a first-in-first-out (FIFO) guarantee in Service Bus. Message sessions enable joint and ordered handling of unbounded sequences of related messages.
Auto-forwarding:
The auto-forwarding feature enables you to chain a queue or subscription to another queue or topic in the same namespace.
Dead-lettering:
Service Bus supports a dead-letter queue (DLQ) to hold messages that cannot be delivered to any receiver, or messages that cannot be processed.
Scheduled delivery:
You can submit messages to a queue or topic for delayed processing. For example, to schedule a job to become available for processing by a system at a specific time.
Message deferral:
When a queue or subscription client receives a message that it is willing to process, but due to exceptional circumstances within the application, such processing is not currently possible. The entity has the option to and can defer retrieval of the message to a later point. The message remains in the queue or subscription, but it is set aside.
Batching:
Client-side batching enables a queue or topic client to delay sending a message for a specified period. If the client sends additional messages during this period, it transmits the messages in a single batch.
Transactions:
A transaction groups two or more operations into an execution scope. Service Bus supports grouping operations against a single messaging entity (queue, topic, subscription) within the scope of a transaction.
Filtering and actions:
Subscribers can define which messages to receive from a topic. These messages get specified in the form of one or more named subscription rules. For each matching rule condition, the subscription produces a copy of the message, which may be annotated differently for each matching rule.
Auto-delete on idle:
Auto-delete on idle enables you to configure an (idle) interval after which the queue gets deleted automatically. The minimum duration for idle interval is 5 minutes.
Duplicate detection:
If an error occurs that causes the client to doubt the outcome of a send operation, duplicate detection helps clear that doubt by enabling the sender to re-send the same message, and the queue or topic discards any duplicate copies.
Use Case:
A Utility bill payments company wants to build an application, which will do the mobile recharge for end users. The application should support all the operators available in the market. The flow is — the user sends SMS in predefined format and application should parse the message and do the recharge accordingly.
Now, the foremost challenge is that the application is unable to process the large number of recharge requests it gets during peak hours. The company is looking for a solution where all the requests are parked as they come and processed one by one.
To meet this business requirement, use the queue mechanism where all the requests are added in queue, and on another side, the queue listener will process the message by reading from the queue.
Azure Storage Queues
Azure Queue storage is a service for storing large numbers of messages. Messages can be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message size can be up to 64 KB. A queue may contain messages up to the total capacity limit of a storage account.
Where to use storage queues

Create work backlog to process asynchronously
Pass messages from an Azure web role to an Azure worker role

Use Case:
A content management system application where files are uploaded by several users daily. The files need to be processed asynchronously by a background process which filters out the media files and sends them for transcoding. When the users upload the file, it gets pushed to queue, and the background process will process the files from the queue asynchronously.
When to use Storage Queue and Service Bus Queue

Storage Queue	Service bus Queue
Your application should store over 80 GB of messages in a queue.	Your solution should be able to receive messages without having to poll the queue.
Your application needs to track progress for processing a message inside the queue. This is useful if the worker processing message crashes. A subsequent worker can then use that information to continue from where the prior worker left off.	Your solution needs the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.
You require server-side logs of all of the transactions executed against your queues.	Your solution must support automatic duplicate detection.
No role-based access supported.	You deal with a requirement to provide a role-based access model to the queues, and different rights or permissions for senders and receivers.
No dead lettering in the storage queue	Dead lettering, supported only by Service Bus queues can be useful for isolating messages that cannot be processed successfully by the receiving application
Maximum message size is 64 KB	Maximum message size 256 KB or 1 MB