Microservice Design with SignalR

Question

We have built a microservice architecture but have run into an issue where the messages going on to the bus are too large. (Discovered since moving to Azure Service bus as this only allows 256KB compared to RabbitMQ 4MB)

We have a design as the below diagram. Where we're struggling is with the data being returned.

An example is when performing a search and returning multiple results.

To step through our current process:

1. Web client sends a http request to the Web Api.
2. Web api then puts appropriate message on to the bus. (Web api responds to client with an Accepted response)
3. Microservice picks up this message.
4. Microservice queries its database for the records matching search criteria.
5. Results returned from database.
6. A SearchResult message is added to the bus. (This contains the results)
7. Our response microservice is listening for this SearchResult message.
8. The response microservice then posts to our SignalR api.
9. SignalR Api sends the results back to the web client.

My question is how do we deal with large results sets when designed in this way? If it's not possible how should the design be changed to handle large results sets?

I understand we could page the results but even so one result could be over the 256KB allowance, for example a document or a particularly large object.

Answer 1

There are 2 ways :-

1. Use Kafka like system which support large size messages.
2. If you can't go with the 1st approach (that's appear from your question), then Microservices can place 2 types of messages for response service (1.) If size is small then place the complete message and (2.) If size is more than supported then place message that contain link to Azure Storage Blob which have result Based on message, response service can get proper result and return the same to Client.