KPilot (was Re: Printers)

[Anne Wilson]

On Thursday 22 January 2009 20:39:58 Michael Comperchio wrote:
>
> Anne,  I know what happens when we assume, but can I assume from your
> email address you might have a better idea that the rest of us what's
> coming?
>
I hear some things earlier, perhaps, and I know quite a lot of people that I 
can ask, when necessary.  On this particular subject, I forwarded a message to 
the list on Monday, but looking back, I'm not sure it arrived.  It's title was 
"KDE 4.2 KPilot coming! (how to both be excited and have realistic 
expectations)".  Because it's important to quite a lot of people, I'll quote 
it in full - but be warned, it's long :-)
<quote>
I just wanted to get a note out to as wide a distribution list as possible
to spread some important news about the upcoming KDE 4.2 release and
KPilot's (exciting!!) part in it. If you don't care about KDE PIM, data
syncing, Palm devices, or KPilot, you may stop reading now and I won't be
offended. =;P

For the last 2 years, the talented Bertjan Broeksema and I have spent our
Google Summer of Code months doing some major rework and redesign for
KPilot. You can see our sync algorithm redesign Use Case here:
http://snurl.com/acftl and some UML sequence and class diagrams that we used
here: http://snurl.com/acfwd. Previous to our redesign, each conduit in
KPilot contained all of the logic necessary for the syncing of record-based
data, as well as the transformation, comparison, and resolution of that
data. This means that each conduit did things its own way. We didn't share
any code between the conduits for syncing, and we weren't consistent about
how the data was synced. In addition, NONE of our conduits had the ability
to synchronize category information from PC to Palm. We had inconsistent
smatterings of success going from Palm to PC, but again, we were
inconsistent and buggy. And obviously with this duplication of work,
whenever a bug popped up in one conduit, it likely existed and had to be
fixed (and most probably did not get fixed) in the other conduits. In
addition, we did a very poor job of ensuring that we didn't lose our users'
data, and subsequently, we had a bunch of "KPilot ate my future" e-mails (
http://snurl.com/acgf0) that nobody was happy about.

So one of our most important goals in our KPilot redesign was the safety of
our users' data. We now go to great lengths at the beginning and end of the
sync process to make sure that our record counts match between Palm and PC,
and that we're able to save all of our data successfully. If anything breaks
in the process, we do not commit or save our changes to the Palm and PC
sides.

We also didn't want to keep writing and maintaining "how to sync" logic in
our conduits anymore. This was a major design flaw previously and led to a
lot of rework and problems. So we designed a record-based (since all of our
syncing is from record-based sources) syncing conduit set of base classes
that all of our new conduits inherit from. This means that now all of our
conduits use the same syncing logic and flow. If we got it wrong, we fix it
and now all of our conduits are fixed. If we need to improve the performance
and efficiency (which we do, still), we fix it in one place and now it's
fixed and improved in all of our conduits. This is a much cleaner design and
it means that our implementing conduit classes are now MUCH simpler, have
MUCH less code in them, are MUCH easier to understand, and all follow the
same patterns, meaning the learning curve for debugging, writing and
contributing conduit code just became darned near close to flat. The job of
our conduits is reduced to data transformation, comparison, and record
domain knowledge, as it should be. The syncing logic all happens in the base
classes.

Another nice feature that we added, although it's not completely finished,
is a final "volatility" check in the base conduit classes. At the end of the
sync, we look at how many records we have created/updated/deleted in
comparison to starting and ending counts, and if the changes are too
volatile, we'll ask our users whether they want us to save the sync or undo
it.

Anyone familiar with programming in professional environments should be
familiar with the idea of commit and rollback. Well, previous to KDE 4.2's
KPilot, we had one mode: change live data on the fly and don't bother with
commit/rollback. This made things nice and fast, to be sure, but we also
lost a lot of data that we shouldn't have. In fact, we didn't even have a
way to not lose data, previously. So this was another important design
decision we made in the new version of KPilot. We do all of our syncing work
in temporary storage and only at the end of a successful sync process do we
commit (or rollback) our changes to the real (PC and Handheld) data stores.

We also have a nice new Keyring (http://gnukeyring.sourceforge.net/) conduit
for KPilot which uses our new base syncing classes.  Unfortunately, we
didn't get a chance to finish polishing it off, and thus it will most likely
be disabled for the final KDE 4.2 release. We'd love your help in getting it
finished, though. Bertjan has even written a nice Qt4-based PC Keyring data
file viewer/editor that we'd love to get finished, polished, and released.
Please help!

So there's a LOT of coolness in the upcoming KDE 4.2 version of KPilot. Not
to mention the fact that it's the first KDE4 version of KPilot. But I also
want to set realistic expectations so that we're all on the same level.

This will be the first release of a major rewrite of KPilot, and we have had
very little user feedback thus far (although there have been a few very
helpful folks and I really appreciate them!!!). This means that there are
guaranteed to be as-yet-undiscovered bugs. I have been doing a bunch of
testing and bug fixing, even since the last RC1 release of KDE 4.2, but
there's just no way I can hit all of the use cases we have. Things are
looking extremely smooth and stable and I have not once lost data (yay!),
but I'm sure bugs are in there, so make sure you make and keep a good backup
of your Palm and PC data before trying new software.

Previous to KDE 4.2, KPilot relied on the external data sources to know
about the condition of that data source's records and PC->Palm  mappings,
etc. With KDE 4.2, KPilot now keeps this information itself. This is a good
thing! We had a bunch of problems in KDE3 which were caused by the mapping
being wrong and/or the state being wrong (X-PILOTID and X-PILOTSTAT).  So we
have our own XML-based mapping files now that are kept in
$KDEDIR/apps/kpilot/conduits/<username>/mapping/ and these help us to keep
track of the mappings as well as per-conduit last-synced times, etc. We
currently rely on this mapping to be valid at the start of a sync (which is
also a good thing), but which has one unfortunate side effect that we'll
most probably not be able to get fixed before 4.2 is release: CopyHHtoPC and
CopyPCtoHH don't work as advertised right now. This also means that the
first time you sync with the new KDE 4.2 KPilot, we will do a "first sync",
regardless of whether you've already synced with KPilot so that we can
correctly establish this mapping file, and you'll end up with a combination
of PC and Palm data on both sides at the end.

We're using Akonadi for syncing your data in KDE 4.2! This is also really
good goodness! This will allow us (eventually, once the Akonadi back ends
are written) to sync your Palm pilot with an Exchange data source, or a
Google calendar, etc., etc. The realistic immediate view, though, is that
KPilot is using and relying on a really new Akonadi solution. KDE PIM 4.2 is
still not fully ported to using Akonadi purely, although we have bridge
resources as an interim solution. This means that before you can sync with
KDE's 4.2 KPilot, you need to add an Akonadi resource for each data type
(Calendar, Contacts, etc.), then change your KDE PIM applications to use
that (korganizer, kaddressbook, etc.), and then reconfigure KPilot's
conduits to use those new Akonadi data sources. There are glitches to be
found in all these layers, to be sure!!  In my testing, it looks like you
have to hit reload in korganizer to see kpilot changes to akonadi's cache
after a HotSync, for example. Also, it's a manual process to convert your
PIM data sources to Akonadi resources so KPilot can sync with them. This
MUST be done, since KPilot now syncs only with Akonadi data sources.
Together, we'll be finding (hopefully just a few) bugs in KPilot, the new
KDE PIM code, and Akonadi. The pay-offs for all of this work are immense,
but we need to have realistic expectations about it too.

We have a serious developer shortage in KPilot--both real-person-resource as
well as available-time-resource--and as a result, we were only able to bring
so much forward from KDE3's KPilot into KDE 4.2. We tried to focus on the
most widely used functionality and, to be honest, that which we the KPilot
developers actually use. This means that KDE 4.2's KPilot will not have: the
Knotes conduit (which was badly broken anyway), the DOC conduit (which I'd
love to get working again but we'll need someone to help code it to get it
done), the Avantgo/MAL conduit (does _anyone_ actually use Avantgo
anymore???), the popmail conduit, or the notepad conduit. So, if you
absolutely have to have one of those, please continue to use the KDE3
KPilot, or better yet, help us port your favorite conduit to KDE4!!! What we
do support in KDE 4.2's KPilot is: Calendar conduit (syncs against Akonadi),
Contacts conduit (syncs against Akonadi), Memofile conduit (syncs to a
directory/files on your hard drive), the Time conduit, the Palm database
install conduit, and the ToDo conduit (syncs against Akonadi). These are by
far the biggest heavy hitters (the 4 main Palm apps), but again, if you
absolutely must have one of the others (or a new conduit altogether), please
help us make it a reality by contributing.

Our little First Time Setup Wizard's device detection functionality is
broken.  Actually, it's been broken for a while. But we didn't have time to
try to get it working, so we disabled that functionality. Really, this is
ONLY a one-time setup issue, and there are far better ways to figure out
what connection string you should be using, and they're pretty standard now
("/dev/ttyUSB1" if you're using the visor kernel module, "usb:" for libusb
syncing, /dev/ttyS* for serial devices, etc.).

Previously, KPilot tried to be a "anything that has to do with a Palm
device" solution, and we had built-in viewers and editors for Palm
databases. These were buggy and error-prone, and confusing, since it looked
like KPilot could sync with your Palm, KDE PIM apps, and any changes you
made to the databases through the viewers/editors. Well, no more. We've
removed these viewers and editors to fix a bunch of bugs, to cut down on the
porting and maintenance efforts, and to provide a cleaner user interface.
KPilot is now focused on being a syncing solution between your Palm and
KDE's PIM back ends. If you need a generic palm database viewer/editor
solution, JPilot is much-better suited to that.

Unfortunately, using "usb:" (libusb) to sync your USB palm still has some
weirdness to it. I have not had time to dig until I found the solution, but
even though we do the exact same thing with "usb:" devices through
pilot-link that we do with old-style USB devices (/dev/ttyUSB1, etc.), we
periodically hit a crash when closing libusb sockets. Sometimes this shows
up as a double-free bug down inside libusb, and sometimes there's other
weirdness. Bottom line is this: "usb:" should be fine for syncing your Palm
with KPilot, but you may see crashes after your data has been synced or when
KPilot closes the socket handles to libusb (pi_close, in paricular, causes
this). This is an area I'd REALLY like some help with, since I don't see
anything that KPilot is doing wrong.

One of the challenges we've always had is in getting our users to help us
understand what's going wrong when something goes wrong. In the past, this
has meant that we've had to tell them to get KPilot's source code,
reconfigure it to make sure debugging is turned on, recompile, re-install,
and test again before we could get ANY useful information for as to what
might be going wrong. This presented an insurmountable barrier to entry for
99% of our users and has resulted in a lot of bugs never getting fixed and a
lot of frustration and wasted time, all around. Well, no more. In KDE 4.2,
KPilot now should have debugging (with tunable output levels) turned on by
default. The positive to this is that we should be able to ask our users to
just run their already-installed version of "kpilotDaemon --debug=9" and
send us the output for us to be able to get a much better look at what might
be going wrong inside the wee beastie. The down-side is that we will be
producing more console spew as a result. I've tried to keep this to a
minimum, but I'm sure this will upset some people anyway. For those folks:
workaround would be to run "kpilotDaemon >/dev/null 2>&1". We can definitely
improve in this area and prune down what gets printed by the default debug
level, and I'd greatly appreciate any patches to help do so.

And lastly, as I mentioned previously, we made some intentional design
decisions that favored the protection of our users' data first and foremost.
As a result, some things will take longer in KDE 4.2's KPilot than they used
to in KDE3. Most notably in this area is our "first sync", where we take in
both collections of data (Palm and PC) and match them up and result in a
combined set. This area has not been performance-tuned (we're currently at
an operational efficiency of O(n^2) in our data matching code, I believe,
which is obviously bad), and there are definitely things we need to improve
on (and would REALLY appreciate the community's help on!!!). If you have a
database with thousands of records, then "first sync" (which happens
obviously the first time, but also if the record mappings are invalid) will
take a while and your Palm may time out. Less than optimal? Yes. Slower?
Maybe, the first time. You lose data? Definitely not. Can you help make it
better? YES! =:)

So, in closing, I hope this helps to get all you Palm users excited about
our new version of KPilot. Never before has KPilot looked so good, been as
careful with your data, or been easier to help debug, extend, and help out
with. There will be some challenges, as there are with any new major version
of software (and this is a doozie, all around!!!). But I'm really pleased
with what we have and where we can go from here.

Thanks, enjoy, and please give us feedback!! As always, bugs should be
reported to bugs.kde.org, e-mails should be sent to kdepim-users at kde.org,
and more-or-less real-time help can be found in #kpilot on FreeNode IRC.=:)

-- 
-[ Jason 'vanRijn' Kasper    //  http://movingparts.net ]-
-[ KDE PIM Developer         //  http://www.kde.org  ]-
-[ bash fun -> :(){ :|:&};:  //  Numbers 6:22-26 ]-
</quote>

I take it that Palm owners will benefit first - but this is a subject of which 
I know just about nothing :-)

Anne
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
URL: <http://listman.redhat.com/archives/fedora-list/attachments/20090122/96bde982/attachment-0001.sig>