Data loss of postal addresses between Evolution and Ubuntu One's Funambol exchange/web UI

https://code.launchpad.net/~rodrigo-moya/ubuntu/lucid/couchdb-glib/fix-571286 branch has the fix for lucid. Fix for karmic package coming soon

Instead of changing everything (including an FDO spec) to match the Ubuntu One web site, wouldn't it make more sense just to change the Ubuntu One web site? There are additional Ubuntu One clients in development and so this seems like a lot of changes on the client.

Scott, the main problem is not with the web UI, but with the Funambol component that manages syncing with phones: the web UI part is just a reflection of that. The Funambol data definition cannot be changed, because it interacts with hundreds of different phone kinds out there, that's why we need two fields rather than one for the "street" part of the address.

I changed the description to better clarify this, thanks for asking.

This really should not be an SRU until after there is general consensus that this is the way free desktops are going.

Scott, we need 2 fields for the street information anyway, and we first added a 'ext_street' one, but since the only application following the standard is Evolution, and we had all the phone syncing using already address1/address2, we decided to use those instead. So, the freedesktop standard is now changed to match that. Also, code has been added to both evolution-couchdb and the U1 server to deal with old records using 'street' field only, so it's safe to do the SRU, once I've got the karmic package ready.

As for U1 clients being developed, for contacts there's only Akonadi, and it doesn't support postal addresses yet anyway, AFAIK (I might be wrong).

Harald,

Would you please comment on how this change would impact Akonadi in Maverick?

Well. We just discussed this a bit on IRC and I would like to highlight the following:

a) vCard is an established standard that defines free-form address data without ext
b) as far as we (that is me and a KDE gentoo developer) can see, akonadi will support phone sync via syncml (an established standard), and syncml enforces exchange of contact data via vCard (an even more established standard). (that said I wonder why the fdo spec cant implement vcard to begin with)
c) I do not see how implementations that by design do not have an ext field can support this without actually following the change
d) each mail client and groupware and essentially any addressbook software on the market can handle vcard, and vcard does not have such a silly ext thingy

In a more specific scenario:
http://api.kde.org/4.4-api/kdepimlibs-apidocs/kabc/html/classKABC_1_1Addressee.html
KDE's contact implementation does not implement any kind of ext field so options are:
1. change KDE to support ext
2. not show that data
3. mess with the data to squeeze it into KDE's free-form street label and then extract it again for sync

1 is simply stupid since it basically suggest that every contact application that would want to use coudb/u1 and does not have an ext field, would also have to change around. 2 is really not much different from loss of data. 3 is a major headache and bound to end up in data loss from bugs.

We agreed that the only viable options are:
* Have u1 implement abstraction, instead of doing it for each client - syncml can, so why should u1 not be able to
* fix Funambol

<apachelogger> also I find it concerning how quickly they are on changing the spec
<apachelogger> ...what if a company internally implemented that couchdb stuff using the spec...

Packages for Karmic and Lucid uploaded, waiting for approval

Harald, so KDE just stores addresses in one big field, right? If so, you are having this problem of more-structured data in the fdo spec, since we are already storing city, postcode, street, etc in separate fields. If that's the case then, what are you planning to do? Or are you talking just about 'street' information? If so, the change you need to do is as easy as concatening address1 and address2 in one string when reading from CouchDB, and splitting it (first line for address1, the rest for address2) when writing. That's exactly what the patch for Karmic and Lucid for couchdb-glib does, to support the old API but the new format.

Also, does this interest mean someone is working on Akonadi to make it read from desktopcouch?

> d) each mail client and groupware and essentially any addressbook software
> on the market can handle vcard, and vcard does not have such a silly ext thingy

See RFC 2426, par. 3.2.1, "ADR Type Definition" <http://tools.ietf.org/html/rfc2426#section-3.2.1>:

"The structured type value corresponds, in sequence, to the post office box; the extended address; the street address; the locality (e.g., city); the region (e.g., state or province); the postal code; the country name."

Also par. 4, "Formal grammar":

"adr-value = 0*6(text-value ";") text-value
        ; PO Box, Extended Address, Street, Locality, Region, Postal
        ; Code, Country Name"

> In a more specific scenario:
> http://api.kde.org/4.4-api/kdepimlibs-apidocs/kabc/html/classKABC_1_1Addressee.html
> KDE's contact implementation does not implement any kind of ext field so options are:
>
KDE indeed does:

http://api.kde.org/4.4-api/kdepimlibs-apidocs/kabc/html/classKABC_1_1Address.html#abb3c1e4c0ae5cc9cc0c3f14ad48e604f

re #10 no, KDE does not store everything in one big field, it only stores the street data in that field. What's more is that I do not see how Evoluation does use the ext data visually...
About concatening though, I find it horribly horribly horribly awful. How do you sync that if user changed the ext on his phone and the ext part on his desktop in different ways, I suspect it will be difficult to detach the ext data from the street without any guidance where to split. That said, what to do if the user changed the address data in such a way that the ext data is unrecognizable? Taking the whole data and stuff it into address1 is probably no option because the user might be ext aware and have provided an ext and expects it to show up as such on his phone. I really do not think that glueing things together in the implementation is a good idea, especially if you consider that at some point more implements are around. If there is a common flaw in all of them, they all need to be fixed instead of fixing it once for all.

And yes, someone is (or in this case will be) working on Akoandi support https://wiki.kubuntu.org/GSoC/2010/HaraldSitter
+ I think you should keep those options in mind eitherway.

re #12 I seem to have missed that. Anyhow, the problem is that no UI exposes the ext field, most likely because the addressee class does not directly expose this interface. Time to poke upstream I suppose...

I do not want to accept this SRU. Changing data storage formats post-release is a no-go area for me. I think we need to work around this in evo/funambol and just support both the old and new data format. (It's also pretty much a "no way") for further Ubuntu releases).

The current transition code is in couchdb-glib, but how can we be sure that there are no other applications which use those address fields in couchdb without couchdb-glib?

This had been added to https://wiki.ubuntu.com/LucidLynx/TechnicalOverview, but needs to be documented in https://wiki.ubuntu.com/LucidLynx/ReleaseNotes instead. The note that had been written there was:

 * Street address information might get lost when editing/viewing contacts in Evolution, the Ubuntu One contacts web interface and synced phones. (Bug:571286)

This needs to be expanded upon to more clearly explain to users what steps they should take to work around or avoid this bug.

rejected upload from lucid-proposed and karmic-proposed unapproved queue in line with pitti's comment above

I already explained to pitti why we are doing it this way. For Maverick we are changing the on-disk format anyway, and these packages I uploaded prepare Karmic and Lucid users for the change, while also fixing the bug they are already seeing, which is that street information gets lost if they edit contacts in evolution and see them in the web UI and viceversa.

So, with these packages having been rejected, karmic and lucid users won't ever see this bug fixed.

I think the reasons this is a bad idea were pretty clearly explained. It would be a good idea to consider them and not just to press on blindly.

Scott, I haven't heard any reason this is a bad idea apart from "changing the on-disk format is not a good idea" and "how can we be sure there are no other apps using it", because the KDE discussion is out of context, given KDE does not use this couchdb database.

So, about changing the on disk format, I understand that from a stable distro point of view, it sounds scary, but we are going to have to do it several times, as we add new fields, like adding photos for contacts, etc. That's why we provide a detailed field-by-field API in couchdb-glib and are adding one for desktopcouch (see lp:~teknico/desktopcouch/contacts-detailed-api/), so that apps don't have to deal with the on-disk format changes. So, changing it in Evolution is impossible, since Evolution uses that couchdb-glib API and thus doesn't know about the on-disk format at all.

About apps using the contacts database, we have Evolution, which doesn't need any change at all given the above mentioned reasons, and macaco-contacts, which is been changed as soon as this fix goes in, and which is not even part of Karmic or Lucid.

@rodrigo - there are several strands here that interact.

* end users write apps to code in the distro. So when we offer them a release, they know they have a *fixed* basis on which to use the API's present in the distro. Anything which would break a third party user using a shipped API is something that an SRU cannot do except under very specific circumstances - such as 'only way to fix a local root exploit' or 'there is no way to use it without this fix anyway, because it depends on server XYZ which has changed its protocol'.

* Code changes are risky, SRU's are meant to be auditable and understandable; on disk format changes to a database sound (to me) to be unlikely to be easily audited and understood. I may be misunderstanding here, but I'd be expecting the pure distro folk to be hearing what I'm hearing here ;)

* It's not clear whether anyone can directly depend on your on-disk format or not - I mean, you have an API, and you have a disk format, but can anyone open up one of the couchdb's with couch itself, and query? if so, then you may have third party things you don't know about starting to use the shiny you've provided.

Forgot to talk about 3rd party apps, so here it is.

First of all, we don't know what 3rd party apps are using it, so not sure if there are any at all. Then, we haven't yet agreed on API/on-disk format stability yet for this contacts database, it's still an unstable API/specification, so there is not API/on-disk format stability assurance for this that 3rd party apps can rely on. If a 3rd party app uses this, it already knows (or should know) that this is unstable and can change.

Also, let me explain why we need, not only for this change but for other on-disk format changes we are going to do, the karmic/lucid packages to be upgraded. Whenever we deploy some change on the server, users in those distros will get the new database format automatically on their next synchronization between U1's couchdb and their local couchdb. So, if we are not allowed to do on-disk format changes, this means we need to freeze the format used in Karmic for years and not improve / evolve it at all. Provided, as mentioned above, this is an unstable API/spec, that would mean having to commit to a work-in-progress for years. So this is not about changing on-disk format for an old distro, it is about making it support stuff that they will be getting from the server, which is not tied to any distro, for the next few years.

You do know there's a KDE client for Ubuntu One being developed that's expected to land in this cycle? It's true that there's no KDE user of the database today, but that's hardly the only relevant data point.

The problem isn't with adding new information to the database, it's with modifying the existing address format.

I thought it was quite clear that the correct answer to the address question was for your web app to convert between the two formats. Patching all potential users of address data to match the format used by your proprietary back end is not the solution.

Rodrigo Moya [2010-05-26 8:56 -0000]:
> So, if we are not allowed to do on-disk format changes, this means
> we need to freeze the format used in Karmic for years

Well, isn't that the point of stable releases?

> and not improve / evolve it at all.

We did not say that, though. The problem is right now that the way we
use couchdb now does not provide any kind of versioning. If there was
a version field which client apps could check, and which would be
bumped on format changes, it would at least be possible for client
apps to handle underlying format changes. Right now it's not possible
at all, and if you mix different couchdb sources from different
releases/sources/etc. you'd just create a mess. As long as this point
does not get addressed, I do not want to accept such kind of
unversioned on-disk format changes as an SRU. Once there is proper
support for versioning, this can be revisited, of course.

Additionally, I still think distro unique MUA patches to change the data format are a mistake.

Bumping to low as the solution is still unclear.

rodrigo_ | pitti, bug #571286 was an issue for karmic, I think, it's not anymore

Marked invalid since Funambol is no longer supported.

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".