Creating scalable RPA processes with Queueing

Often when I am talking with customers and prospects they ask about the concept of queueing transactions and then level loading across multiple agents/servers/bots so that their workloads are instantly scalable up or down as needed by adding additional agents into the mix.

Here's an example where queueing might be useful: Let's say a company needs to interact with a vendor or customer web site, an SAP application or some other UI based ERP or business application that can only process one transaction at a time. Often a single bot won't cut it because it's not fast enough because of application or other response time issues, so you need something that is infinitely scalable. Also a server in many cases is limited to one transaction at a time because it's automating using a user interface. Background processes are another story because a single server/agent/bot can handle many background process threads simultaneously.

By intercepting all transactions as they come in and funneling them to a single queue or inbox, one or more (pick a number) robotic processes can pick up the next transaction as it finishes the previous one. Think of it as being similar to picking up the next phone call in a call center or being next in line at the grocery store. First-in, first out using as many lines available.

So how does this queueing thing work ? Well the idea is that as a transaction comes in and gets added to a database or some other data store, a queue entry is written immediately so the next agent/process/bot can pick up the transaction and get it done. As the next sequential entry gets picked up it gets removed from the queue or marked as in-process or complete.

You might consider using a message queue system such as Microsoft Message Queue (MSMQ), WebSphereMQ, RabbitMQ, ActiveMQ or even a platform such as Apache Kafka. There are probably others that I missed. These queueing systems can be implemented simply and used with any RPA tool set that can call an API or web service. However they do add another layer to the complexity and overhead of your RPA process by adding another moving part for your application or RPA process to monitor: The message queue service.

Another simpler option perhaps might be to write queue entries to a database so that they can be processed easily and also the queue can be visualized via an SQL query as needed for reporting purposes. If you have read articles on table queueing theory on the web, there are lots of things written that recommend not using databases as queue mechanisms. I have researched a lot and found that SQL Server (and probably other databases as well) can work nicely as a queueing system if you're careful. The trick is when reading an entry, how do you make sure you process a particular database queue entry only one time ?

While researching on SQL Server methods for locking I learned about the READPAST and UPDLOCK flags. I won't regurgitate other articles so you can simply google the highlighted keywords for yourself and draw your own conclusions. However the upshot is that when these two flags are used on a query combination that reads and updates/deletes a record all in a single transaction, you should be able insure that each queue table entry is handled correctly and only once. When writing a new entry to a queue table you simply perform a standard record insert, so feeding the queue a bunch of transaction data that needs to get processed is very simple.

Hopefully you'll find this technique to be helpful when you're scaling your RPA, Workflow other process automation workloads. This technique should work with any automation toolset or custom process you create where queueing is required.