I have a queue of messages representing filesystem operations that need to be processed in order. A message may succeed or fail when it's sent. A change message, for example, is generated when a user creates or saves a file.
- Delete Foo.txt message sent
- Delete Bar.txt message sent
- Change Baz.txt message sent
- Change Foo.txt message sent
In the most common case, I pop the top item from the queue and send it off for processing, which succeeds.
But suppose this happens:
- Delete Foo.txt message attempted (FAIL)
- Delete Bar.txt message attempted (OK)
- Change Baz.txt message attempted (OK)
- Change Foo.txt message attempted (OK)
What should I do in this case?
I want to move on and process the next item in the queue, but it's not that simple. In this case, "Delete Foo.txt" must be processed before "Create Foo.txt", otherwise we're deleting Foo.txt from the search index (see "more details" below), even though it exists.
One approach I had considered: a key-value store, where the document path is the key, and the Message is the value. In the context of a filesystem, the only operation that matters is the most recent, so overwriting the Message associated with a file if it hasn't been sent might work.
What kind of data structure and algorithm is right for this situation? I don't want to reinvent the wheel.
A few more details:
- New items are constantly arrive in the queue, from multiple producer threads
- The messages are consumed from the collection by a single consumer
- A message is a serialized object, sent to a RabbitMQ broker
- This queue exists in case the RabbitMQ broker goes down or a Publisher Confirm fails
- Messages are consumed from the endpoint by a different service which performs various actions to keep a search index incrementally up to date
- C#, if it matters
I am sending messages about filesystem operations that have occurred. These messages are used to keep a search index (elasticsearch) up to date. A few minutes behind is perfectly acceptable. If a user makes three saves in rapid succession, I don't need search index updates for the first two saves. (If it happens, that's fine, but it's not required.)
CRUD operations are done by users working with their files on a busy network share. Each write operation triggers a message that has to be sent to a persistent work queue (the RabbitMQ broker). An example message might contain data that boils down to "File F has been deleted" or "File Y has been changed". (No file contents are contained therein.)
(These messages are consumed by another component downstream that performs an action based on the contents of the message, which is outside the scope of this question.)
Because messages represent filesystem actions that were performed in a sequence by users they either:
- Have to be sent in order so they can be processed in order OR
- The message associated with the most recent write operation on File F should overwrite any queued message associated with File F that hasn't been sent yet
In my example above (second unordered list), it's safe for the message representing "Change Foo.txt" to overwrite a failed "Delete Foo.txt", because the Change message was triggered after the Delete message. (The time where Foo.txt didn't exist isn't important.) Hence my thought of a using key-value store of as a holding area in case of a Message send failure.