Completelly remove the use of design docs from couch backend
The use of couchdb design docs views and lists can be slow for some reasons:
- views have to be updated for each document change.
- for each query to a view, the whole list of documents is traversed.
- data is passed to a javascript interpreter running in another process and back.
Those reasons also make it more complicated to compare views with plain document gets, because:
- views return many/all documents, while a document get returns only one document
- the time taken also depends on the application use of the view (frequency of use, amount of data retrieved on a view, etc)
- view results are cached on a tree which is efficiently updated on doc change.
Despite that, this is a simple time comparison between normal document gets, view functions queries and list function queries (using the most simple views/lists possible):
get 100 docs finished: total: 0.408011 secs mean: 0.004080 query 100 views finished: total: 1.094964 secs mean: 0.010950 query 100 lists finished: total: 3.280392 secs mean: 0.032804
So the use of a view function is 2 times slower than a get, and a list function use is 8 times slower than a get.
Another comparison worthwhile is between view/list functions and the "_all_docs":
get 100 docs finished: total: 0.393654 secs mean: 0.003937 get _all_docs once finished: total: 0.007916 secs query 1 view with all documents finished: total: 0.182986 secs query 1 list with all documents finished: total: 0.032574 secs query 1 view with all documents finished: total: 0.007209 secs query 1 list with all documents finished: total: 0.038157 secs >>> insert one document, force view update get _all_docs once finished: total: 0.007655 secs query 1 view with all documents finished: total: 0.019355 secs query 1 list with all documents finished: total: 0.031298 secs query 1 view with all documents finished: total: 0.006805 secs query 1 list with all documents finished: total: 0.037628 secs
From the above we can see that:
- the first use of a view function is 25 times slower than subsequent uses.
- the update of the view function cache is ~3 times slower then subsequent uses.
- the use of list functions doesn't seem to be affected and always take ~5 times more than view function.
- the view function time is comparable to @_all_docs@ time, after the view has been updated.
All this means that if not used with care, view functions can lead to problems.
There is at least one other reason why we want to get rid of design docs:
- Currently, we use timestamps to insert u1db transaction information inside the document structure. By using view functions to list those transactions and by sorting them by the timestamp, we retrieve a list of changes to the database. The advantage of this approach is that we can save both the actual document content and it's transactions in an atomic opration. The downside is that it is very prone to problems: if the timestamp of the system changes for some reason then the database metadata will get into an inconsistent state and that will be a problem for future syncs.
Because of all this, it seems to be a better option to simply remove the use of design docs as a whole and go for simpler python-application-side solutions, even though they might also not be the most elegant solutions.
(from redmine: created on 2016-07-27, closed on 2016-08-13)