Please support redis as db

Can you explain why a database is needed, and why redis in particular? I think Roger removed the db dependency a few releases ago and it certainly makes mosquitto much simpler.

I think mosquitto need to store the clientid, messages to some place. Redis is a in memory db system, it is quick and simple. So if redis is supported, that will be great!

Could you give more of an idea what you're trying to achieve?

1. I just want to manages all of the clients and messages in memory, i don't want to read or write disk in most operations for clients.
2. the in-memory db system should easy to use, easy to learn.

I'm still not sure what you actually want to achieve. Could you explain an actual scenario where you would make use of this feature?

I'd like to broaden the discussion while we're on this topic of databases.
AFAIK the current persistence implementation is simply of a (periodical) dump of the memory into a flat file.
So the mosquitto server has to keep all the information in memory when running, right?
While it's perfectly fine with a few hundreds messages in the queue, that clearly doesn't scale if the server has to keep many more messages, for many more clients.

More than the use of a specific database library, I think that's this all-in-memory behavior that should be improved.
Be it with a mosquitto-specific database implementation or a more common SQLite or Redis.

In Mosca (https://github.com/mcollina/mosca) I support MongoDB, Redis, LevelDB (embedded) and Memory.

Even though implementing multiple DB support is not simple, I would suggest you give LevelDB a try. It is very fast.

Glad to hear that there exists other MQTT broker implementations.
However I'm running in a constrainted environment (OpenWrt router), so I think node.js is too heavy for my use case.

Thanks for the pointer to LevelDB. It looks very interesting and maybe even more appropriate than SQLite.

Depending on the resources of your OpenWrt router, you might want to give it a try.
It works very well on a Raspberry Pi.
http://oskarhane.com/raspberry-pi-install-node-js-and-npm/

LevelDB is very suited for that kind of task.

Roger, so what do you think of supporting various DB backends, in particular LevelDB?
Or even a home-made database implementation but which would not be memory limited.

Using redis/leveldb for persistence should be fairly straightforward. Using them to replace the in-memory data is much less straightforward and would require a lot of work to do properly. I used to use sqlite as the database backend, but the performance was poor due to the complex regular expressions required to do proper topic matching (see the notes at https://bitbucket.org/oojah/mosquitto/src/tip/doc/historical/old-regex.txt ). Moving to the current system produced hugely significant performance improvements. I doubt that a key/value store would be as bad but it would need to be designed carefully.

Although mosca supports redis as a persistence it does not implement topic/subscription matching very well so subscriptions will not work properly. Matteo, you should look at file mosquitto/test/broker/03-pattern-matching.py for some examples that fail in mosca.

Additionally, I don't believe that the regex I linked to is correct. It needs to be even more complex to work properly.

We do not use a regex at all: we use a data structure called 'trie', the
same thing RabbitMQ uses.
Our approach is mixed: we keep an in-memory data-structure with the list of
the 'offline' clients, then we store only the packets on disk. Here it is:
http://www.rabbitmq.com/blog/tag/trie/.

Thanks for the failing tests, we will fix them. The whole pattern matching
is so hard :/.
The fact that there is no 'official' test suite is very painful.

Il giorno giovedì 11 luglio 2013, Roger Light ha scritto:

> Additionally, I don't believe that the regex I linked to is correct. It
> needs to be even more complex to work properly.
>
> --
> You received this bug notification because you are a member of Mosquitto
> Development, which is subscribed to mosquitto.
> https://bugs.launchpad.net/bugs/931028
>
> Title:
> Please support redis as db
>
> Status in mosquitto: an mqtt message broker:
> New
>
> Bug description:
> Please support redis as db
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mosquitto/+bug/931028/+subscriptions
>
> --
> Mailing list: https://launchpad.net/~mosquitto-devel
> Post to : <email address hidden> <javascript:;>
> Unsubscribe : https://launchpad.net/~mosquitto-devel
> More help : https://help.launchpad.net/ListHelp
>

On Fri, Jul 12, 2013 at 9:02 AM, Matteo Collina <email address hidden>wrote:

>
> Our approach is mixed: we keep an in-memory data-structure with the list of
> the 'offline' clients, then we store only the packets on disk.

Yeah, that's the architecture I'd very much like to see in Mosquitto.
Currently the on-disk persistence feature caters for data safety, which is
nice.
But the memory usage increases too much for environments where memory is
scarce.

I'm putting this on the list to look at for 1.3.