KGiMAP

A not so beautiful Allston morning turned out to be a rather useful one with the slashdot post [ Free IMAP On Gmail]. I checked my gmail settings and (apprently I am among the fortunate ones here) I saw the "Enable IMAP" setting there. Wee...

Fired up Kmail, added a new account for Google as a disconnected IMAP. And on my way to sync ... I was curious how the labels were going to be handled, I have about 15 of them based on which mailing list or which bugzilla an email is from (I have mentioned this before - I prefer organization over search). It was nice to see the labels getting downloaded as imap folders. And there is a special folder [Gmail] which contains the regular gmail folders - All mail, Starred, Trash, Spam, Sent Mail. Thats when it struck me!

All the emails are going to be duplicated or triplicated or could have even more copies. Labels are not disjoint, so there are mails in my Inbox that are also in beagle-bugs and of course all mail ever is also in All Mail. To add to the trouble, deleting an email locally is going to archive the email (basically, it removes the Inbox label from the email) and not delete it. The huge amount of mails in the Spam folder is not healthy for my hard disk space. The huge amount of emails my beagle-bugzilla filter sends out to Trash is similarly bad.

As of now I resolved the situation by asking all my filters (I have one filter for each label) to archive the email ( Skip the inbox) and then apply the label. I asked KMail to not subscribe to any of the [Gmail] folders. And I created a Gmail label(IMAP folder on KMail) Totrash which I marked as my Trash folder in KMail (for the GMail account). Now whenever I delete a Gmail email from KMail, it will go to Totrash which I can empty periodically from the web-interface. What a kludge! I wish there was a rule to empty folders or a filter move emails from Totrash to Trash!

There is no third way

(This post is not about beagle though it is hosted on PlanetBeagle.
However it is somewhat related to "search".)

I have seen two approaches to finding information, in land of software
as well as real-ware. There are people who find it themselves and
there are some who use an external agent to get what they need. Human
memory is expensive, so these are the most obvious approaches. I have
seen equal number of people on both sides.

In the first group are people who like to keep everything organized,
from physical files to computer files, books to RSS feeds, knives to
applications. Organization typically involves grouping the items into
categories and re-grouping again if the number of items in a category
grows too much. Its a pain to remember things, then there is so much
to remember. Care is taken so that the number of categories don't grow
too large or the nesting isn't too deep. Computer Science people
cashed in on such problems, analyzed them to death and presented us
with databases, shallow and deep trees, hashtables, sorting etc.
Whatever the approach, the objective remains that reaching one
particular item should take as less amount of memory and labour as
possible. Memory and labour appear as trade-off here, so there are
variations of this where the most frequently accessed items are
optimized more than others (compression, source coding, amortized
algorithms, cabinets with upper and lower shelves etc.).

The second group of people use an external agent to do this task. In
reality, not everyone belongs to only one group, people tend use
whichever approach is suitable. Anyway, the external agents could be
one's mom, secretary :), a database, a filesystem, search engine etc.
They are nothing but efficient members of the first category. The
efficiency matters here; tolerance gets extremely low if I am not
doing something but hiring someone to do it.

It is clear that when dealing with items on the start menu, the same
ideas would come into action. OK, so the rest of the post is about
start menu, you can stop reading now.

Either we have a menu, properly organized or we have a dump of
everything with an extremely efficient way to get what we want. OK, we
can do the optimized variation of the first and add search to it to
make it work for the second group (e.g. kickoff, internet with search
engine and directories, filesystem with desktop search). But to get it
done right, both should be done with precision and care. Which means,
the search has to be fast and accurate and categorization should not
be too deep nor too flat. Geeshh... that is difficult.

Actually not that difficult for applications. If done the right way,
they can be arranged in a start menu with categories (not Windows
style though!). Or they can just be there and you find them using a
search tool (quicksilver, katapult, deskbar). Or both (kickoff,
kickoff spinoffs).

This is where I found Mandriva to go wrong. They had a nice start menu
to begin with (because Mandrake had a nice start menu to begin with).
In the latest distribution, the Mandriva specific categorization has
been replaced with a two level (i.e. top-level with
network/tools/office and then apps in them) menu. Some of the submenus
became too large, and they have been put behind another nesting called
"More" (e.g. "Firefox" went behind more because it not native KDE).
The end result is a menu, which is only two level and pretty flat (so
less to remember which submenu to go to) but once in the submenu, have
to memorize the position in a pretty long vertical list. Add to that
the effort in dragging the mouse a significant vertical distance. I
have entries that requires me to drag the mouse all the way from top
(I have my start menu button at the top-right corner) to the bottom -
that's 1024 pixels. The earlier menu was the same with one more
nesting, like "Browsers", "Spreadsheet", "Terminals" where I only had
to memorize the levels and then look in the 2-3 items in each
immediate category. The vertical distance covered in the current menu
is larger than the horizontal distance due to that extra submenu.
Adding to the woes is the lack of an embedded search option.

There is another aspect with menu which make vertical movement much
harder than horizontal movement. Its related to Fitts's Law. But first
a small digression. I like to call it Fitts's Theory. Because it
describes certain operation in a particular model. Calling it law
isn't wrong, except it makes it sound like anyone not following the
"law" is from Mars. Once again, it tries to model some observed
phenomenon, and does not aim to dictate how the phenomenon should take
place. Unfortunately most commentaries about Fitts's law seem to think
we should all abide by Fitts's law or leave this planet.

Anyway, in menus the entries are horizontally layered. The menus are
designed to open a submenu when the mouse in on the corresponding
entry. The problem happens if I try to follow a straight line from an
entry to an entry in a submenu. If the entry in the submenu appears
significantly lower in the submenu, the mouse will inevitably cross
over the next entry in the menu, which will open the submenu for the
next menu entry thereby forcing you to start again. Vertical motion
isn't unrestricted. It happens to me always when I quickly try to go
from the start menu to some application whose position I know thereby
trying to go straight from the button to that entry. If I am not
careful, I end up opening the submenu of the next menu entry in a
haste, then move left and up to find the right menu entry, then move
right _carefully_ to horizontally enter the correct submenu and then
rush down to the memorized spot. All in a wink of an eye and it has
become my second nature now. But every time this happens, I feel
slightly irritated. If menus are designed to take advantage of the
horizontal freedom of movement more, it would be much smooth. Deeper
nesting has the mentioned side effect. By the way, radial menu or
submenus that open in the middle or in the first entry should be
better. But apart from a few radial Firefox context menus, I have not
found any other implementation. I love radial menu.

That is all I had to say about start menus. Before finishing, I should
point out that even for the first group of self-organized users,
having someone or something to search is not totally out of the
question. Some form of data are too vast to organize (bookmarks and
directories wrt internet), some have no single way of organizing or
some even have no of meaningful grouping criterion. And even if you
arrange all your computer files in directories, emails in mail
folders, feeds in feed reader folders, websites in bookmark folders,
if you are to find this blog (containing "Fitts's Law" but unrelated
to beagle), then good luck!

I saved 80MB

Its a common question in the database community where to store large binary
objects, commonly known as BLOBs (binary large objects). There are two
obvious options, store it in the filesystem or store it in a database. And by
twisting arms we get the third obvious approach, a mix of the two. There are
some studies in database academic community on the tradeoff.

Thinking carefully, the database and filesystem work in a similar way when
storing objects. Both store data in tree fashion, providing fast access to
any object. Both can add or delete object. And both suffers from
fragmentation.

The fragmentation is the key issue in my test case. So what is my test case.
It consists of lots and lots of small files and I mean, really small files
and some very large files. Just to give an example, I have a total of 20000
files. Of them about,
500 are above 4K
1000 2-4K
2000 1-2K
16000 less than 1K.

There is a reason for this extremely skewed distribution. The files are
textcaches in my home directory. The main contributing factors are about 8000
files (a mix of source code, tex, pdf and html and oh ... lots of pictures)
and 19000 emails (without spam). Now source codes are self caches i.e.
separate textcache is not stored for these files. Same with txt. Pictures and
other media has no text to cache. So the bulk of cached text comes from html,
pdf and office documents. Most of such documents in my home directory are
generally pretty large i.e. with a significant amount of textual content.
This is in sharp contrast to the behaviour for emails. The textual content
from the emails are stored. Also, unlike the html files on the disk, the HTML
parts in these emails are small, really small. Those contribute to the huge
number of extremely small files. If you are wondering how come I got 3000
absurdly small 20 byte files, I store the gzipped copy of the extracted text
in the textcache.

So now the question is how to best store data with such distribution. The
simplest thing is to use the filesystem. Filesystem are equipped with
sophisticated methods to avoid fragmentation. But with a fixed block size
(like 4K on my home directory partition), there is bound to be some
fragmentation when storing small files like above. On the other hand,
historically databases were concerned with storing small strings (with all
sorts of querying and update speedup; and they are pretty good at that).
Recently, most of the databases have added support for large binary objects.
There is also a small twist to this problem. The data is mostly static but
there are deletions (and sometimes modifications). The last important factor
is I am using the embedded sqlite database which stores all its data in a
single file; so in essence, what I get is a "fast retrieval and
update"-enabled huge tar file.

The prevailing folklore is that database is better for smaller objects and
filesystem for larger objects. What is small and what is large depends on the
nature of the problem at hand. There are some papers doing detailed analysis
of the tradeoff, and they conclude somewhat along the same lines. I found one
paper showing that adding and deleting lots of large objects over time causes
enough fragmentation in a database that it can lag behind a filesystem.

I am not much of an experimental person myself. With the above testcase, I did
not even think twice about what I should do. Well, for other reasons (like
heap fragmentation in my application, since reading from a database would
require me to store the entire data in memory) I was a bit biased towards the
filesystem approach. So I decided to take the middle path of a mix of
filesystem and database. Thanks to decent APIs, implementing the hybrid
approach did not turn out to be anything complicated. I store the first 4K in
a buffer and then if I get more data, move everything to a file otherwise
save everything in the database.

All the 19500 files with less than 4K size are now safely sleeping in the
database, which would have otherwise taken 4K each.

(* The title is not entirely correct. I did not have to waste 4K blocks for
each of 19500 files, but now my sqlite database is 26MB instead of something
much smaller. But I am confident, this is good for the mankind at large and
for the dolphins too.)

Tracker-ing looks fun

Jamie checked in thunderbird support to Tracker today. How do I know ? Because I subscribe to svn-commits-list and my gmail filter separated him out when he was adding the beagle thunderbird backend (probably with some modifications, haven not checked the changeset) to tracker.

This is looking pretty cool. I also found out Tracker-0.6 was released with some extremely useful features and a damn good UI. The UI groups the documents in an intuitive way, shows more context information from matches, allows you to do tagging inplace (and has gradients, uh-on). The core daemon (is that what it is called ?) can scan evolution mails, and pidgin logs. Now it can also do index thunderbird mails. It lists XMP metadata support too - which is nice and something I never managed to finish in beagle.

I am pretty sure people will find it useful. I could not resist myself to admire it and share it with others. Well, I was told that tracker is set to become the default desktop search engine in ubuntu, so a lot of unhappy souls are about to become happy soon.

For me, I will continue using beagle. Why ? Because, it does what I need it to do. And if it does not do something, just remember it is written in C# :). Meanwhile, you make your own decision.

How is WebBeagle for a name ?

(Or FireBeagle ?)

More...(drag mouse)

Close Tab

Forward

Next Tab

Reload-Stop

Close Tab

Back

Home Page

I always wanted to search beagle using a web-browser. My desktop searches are mostly infrequent but complicated. So if I am unable to find something I definitely need an index-based search tool (e.g. beagle) but on the other hand I don't want to keep an application open on my desktop for a long time (beagle does not consume significant memory for me, so it can keep running in the background). That was one reason why I wrote the kio slave for beagle (I have stopped liking it for a long time) and followed it with a kde deskbar like applet (which is nice but with limited functionality). Now that I am done with shameless advertisements, let me share with you yet another way to query beagle.

Today I wrote some ajaxy,xslt-ed webpages to allow users (read: me) to query beagle using a browser (err... firefox, konqueror-3.5.5 does not have XSLT processor). It uses the networkservice backend that can be used to query beagle over network (based on last year's Google Summer of Code projects).It uses internal knowledge of how queries are serialized, lists all the information in a boring way and does not show snippets (yet). But it works and was reasonably fast in displaying 42 results. Besides the boring UI, allowing browsers to access services always opens up some security hole, so it is disabled by default. If you so desire, use it but at your own risk (check the commit log for how to turn this feature on).

I do hope to get this feature properly implemented. The things need to be done are:

1. Fix the network-backend (it is sufferring from some crashes).
2. Probably related to the network backend as well, some kind of search authorization is needed.
   
3. Use CSS + javascript for the results page to group/sort the results, and make them look decent.
4. Get snippets and display them. I am thinking of retrieving snippets only on demand. Somehow the name and the location of files or the sender and the subject of emails help me more in filtering out search results than snippets. Which is quite unlike how I use web search engines.
5. Figure out a way to use the C# or libbeagle API to create the xml request messages. Currently they are hardwired. If the solution turns out to be too complicated, it might not be a bad thing to leave the format hardcoded as it is now.
6. More cosmetic, separate command line and configuration options for this feature.
7. This is more related to the network query implementation; figure out how to use the QueryDomain thing meaningfully. The results from some backends only make sense on the same machine and even for some of them, it is tricky to open the applications just from the URI itself. So does it make sense to show e.g. evolution mail hits in the browser ? Does it make sense to return gaim (pidgin) hits when queried over network ?
8. How does the browser behave (read: choke) when it receives 1000 results (100 results each from 10 backends)? (i.e. the DOMParser has to parse a huge string and form a huge DOM of 1000 Hit nodes. You see why I dont like to get all the snippets beforehand :)?
   

Hey Firefox, Beagle This ... NOW!

Consider this scenario: B was writing an article when he remembered that he read a relevant blog some weeks ago but did not bookmark it due to his lack of foresight (stupid B). What to do ? He could possibly run a grep in his Firefox cache folder hoping that the cache would still be there or scan his FF history for a familiar looking URL. Anything better ?
If B was using the FF beagle extension, he could find the blog instantly

Really! Then I will pose this requirement. B has been using the new FF beagle extension for some time now. But he does not want to index all his browsing history. Sometimes he just wants to browse off the record. Maybe after a while he wants to index the pages he visits. Pretty random ... but hey, that's B's nature.
No problem. The new extension puts a small status icon, which can be  used to disable/enable the automatic index; really, at B's own sweet will.

That was easy. How about this one, a bit harder. Someone told B about FF beagle extension, and B started using it. But this time he needs a PDF paper he found online. He does remember that he read the paper and he must have read it in the browser itself. Now what ?
If B was using the new FF beagle extension, he could even search for embedded documents. Embedded content is not indexed automatically (as of now) but can be indexed as required using the status icon menu.

Hmm... But what if B visits a page with auto-indexing disabled and then decides that he better index the current page. Now what ? Does he have to enable the auto-indexing and then reload the page. Thats stupid!
Heh. You gotta be kidding. Of course he can use the status icon menu or the page context menu to index the current page.

If B wants to index an image, or take this one, sees a link to something (PDF, image, java source file, mp3 ...) and wants to index it without loading it explicitly ?
The context menu provides options to index an image and index a link; in both the cases the file is retrieved in the background and indexed. All filetypes that beagle can filter, can be indexed using the index this link feature.

This new extension seems to uber-useful. I guess I err... B should really start using it. Can B save notes while indexing on demand i.e. if he is indexing a link he might want to index a few extra keywords (for querying or for reference) ?
Yeah! B gets that too. If B checks the corresponding preference entry, the extension will prompt for extra keywords to save and index when B explicitly asks it to index something.

I will be good and share this new beagle extension with everybody, written by Tao as part of Google Summer of Code 2007. Soon to be found in beagle trunk.

Mr Beagle, you are on the air

Back in India, I did one good thing. Finished adding Alexis Christoforides' project of Google Summer of Code 2006 Beagle over the network. Its a bit untested with the recent versions of beagle and I didn't read the code completely, so I left it in a separate branch for the time being.

What can you do with this wonderful project ?
- Start beagled on one machine
- Add that machine (with port 4000) to the list of neighbourhood nodes via beagle-config on a second machine in the network
- Remotely query the beagled on the first machine from the second machine

The SoC project was a complete one, with GUI support. However, I only added the basic infrastructure and enabled the cli tool beagle-query to query remotely. I am not sure if querying for emails remotely makes sense, how will the results be opened ? As of now, only the filesystem backend is enabled to send query over the network.

Ideally, beagle should automatically find out other networked beagled in the network. Don't despair, there was another SoC project in 2006, Avahi Integration by Kyle Ambroff doing exactly that. Sadly I am too preoccupied to work on integrating the Avahi work. But with that and the GUI parts of Network Beagle, beagle would be useful in more ways than ever.

Silence of the Dog

Lately beagle releases have slowed down quite a bit; there were a few bug-fix 0.2.16.x release and another 0.2.17 bug-fix release (it was supposed to 0.2.16.4, but the changelog was too large for a point release). The underlying goal is to get ready for 0.3.0; svn trunk is changing so rapidly these days that it is difficult to isolate the simple ones and make them into a 0.2.x release. On the other hand, the changes are too major to be put into a 0.2.x release (they would also need extensive real life testing).

Recently I moved beagle to use taglib-sharp for filtering music files from entagged-sharp. I was told entagged is no more actively maintained and taglib is definitely seeing a lot of rapid development. My timing was not quite right, the 4th March news "Entagged is unmaintained" is followed by the 28th March news "Entagged is maintained". I came to know about it only after I made the transition. Too late! On the plus side, taglib# has support for larger number of formats and is being used by Muine and Banshee, so expect sharing of taglib-sharp libraries. Unfortunately, there are no taglib-sharp packages out there yet (there is a proposal for a debian package), so all the mono apps are currently including taglib-sharp by its source. We too initially source included it, then removed it and instead linked against the package. But if there are no packages for the major distributions, it might make sense to source include it; compiling Beagle is pretty demanding anyway.

In other news, I used the extremely handy heap-shot to identify that instances of IndexReader were not being GC-ed even long after the corresponding method ended. Explicitly setting them to null immediately freed them. I suspect some thread local storage magic happening behind my back. Note to self, set IndexReaders to null immediately after they are closed. Did I say heap-shot is amazing ?!

There are several more improvements to the speed and memory performance of IndexHelper and BuildIndex. One notable feature I added was to reduce re-indexing of files which could not be filtered before. Due to the inherent distributed nature of beagle indexing, the crawler is always separated from the indexer. So if the crawler finds some file which was not filtered before, it has to re-submit it to the indexer. Who knows! There might be a suitable filter now. The downside was that a lot of files were being repeatedly re-tried by the indexer, slowing down the whole process. I decided to store the files containing the filters and their last modified times in a filterver.dat (akin to mozilla pluginreg.dat) and if the filters were not changed since last run, assume that there is no newer filter. Fair guess I would say.

Beagle knew how to index email attachments for quite some time; some months ago it also got the ability to index archives. However all along this was done by extracting the included files to a temporary file and then indexing it. This was done primarily because of the way included content (aka child indexables) were handled and also due to the fact that some of the filters only worked on physical files and not streams. This whole temporary file business never pleased me, there were race conditions which could leave undeleted temporary files in the system, even small included files had to be written to disk and further, extracting the contents of an archive to index it defeated the whole purpose of archiving it. Last week, I added the infrastructure to allow indexing of archives and email attachments without extracting them, if the filter permits of course. The infrastrusture is there, the archive and email filters should be modified to take advantage of this.

Finally, one feature I personally would like to see in 0.3 is support for XMP sidecars. XMP sidecars allow users to add a separate file.ext.xmp file containing arbitrary metadata (but in the XMP format) about file.ext. Really extensible solution for metadata. The main part of the code is in svn trunk; it still does not support renaming or deleting xmp files. Hopefully it will be finished in time.

This will probably be my last post before my annual break to the land of mangoes (" fruit of the gods"). Sadly, I have (knowingly) only tasted about a dozen varieties of mangoes, out of over 300. I will definitely try to increment the number this time. Next post, July.

Upgrade to Spring

Yesterday in a fit of mind I decided to upgrade to Mandriva latest release.
The steps included
- backing up .kde, .kderc, .qt, .gtk* and .local
- logging out of kde
- setting up a mirror as a distribution source for urpmi (mandriva is offering
non-free e.g. sun-java in its free source these days, but I still need the
PLF sources for codecs, BCI enabled freetype and fontconfig and a few other
things)
- # urpmi urpmi
- # urpmi <bunch of> kernels
- # urpmi --auto-select
and then selecting the ones I would like to upgrade from them

Soon I was running Mandriva 2007.1 Spring (Free). Yay! Its beautiful. The
Ia_ora theme and other Mandriva artwork is gorgeous. I legally own a XP cd,
from which I extensively use Verdana (for text) and Tahoma (for widgets).
They look wonderful as always with plf freetype (w/ hinting). I like to use
large fonts, enough to be readable 4 ft afar but somehow the deja or
bitstream fonts have a weird fuzziness in the curves of 's' and 'o's. I
cleared the settings of a test user account and a default new account looks
quite good (apart from the kbfx-styled mandriva menu).

KDE was upgraded to 3.5.6; I was actually using a few kde-3.5.6 packages from
cooker so there was no huge surprise. I was worried that mandriva would mess
up some of my settings when I log in for the first time as my normal user but
thankfully that did not happen. The system feels faster, konsole definitely
starts faster than before. Overall, I am extremely pleased with 2007.1; I
wish them all the best.
http://wiki.mandriva.com/en/Releases/Mandriva/2007.1/Tour

Some minor annoyances:
- mandriva kernel (based on 2.6.17) still has the weird cpufreq bug where
scaling_max_freq is same as scaling_min_freq (thus rendering all the
governors useless). It is probably the same problem described in

http://www.mail-archive.com/linux-acpi@vger.kernel.org/msg04484.html
- Suspend to RAM is broken in mm kernel (Mandriva has moved to pm-utils and it
works like a charm w/ the default kernel)
- tmb kernel has some problems of high CPU usage when copying files and broken
resume from s2ram. I still need to test the other tmb versions. I really like
the tmb kernel improvements.

Theory Guy...

This is not really techie but has a somewhat tech-vs-theory war flavor. The following list came out of today's departmental Friday lunch. Top 10 "ways to tell a theory person from a systems person":

10. One gets job and another does not.

09. One likes 10000n^2 better than n^3.

08. Space-time is important outside Star-trek.

07. In therory they are same, but in practice they are not.

06. P is not equal to NP divided by N.

05. n^10 is efficient.

04. SAT (Famous NP complete problem also known as Satisfiability) is most of the times easy to compute.

03. (Specific to our department) One can be only be found across the street.

02. Its pronounced "Lee-nux" (not "Lai-nux")

01. Computers do more than just email ?!

TinyBeagle or a Lucene Example

Recently I read this interesting comment in an OSNews article. It tried to briefly summarize what beagle is. I take users' comments very seriously and this person seem to know some internal of beagle, so I thought maybe he is true (modulo some factual errors). Maybe the only new thing in beagle is the crawler, the GUI and the scheduler; its mostly little C# glue code tying up a few third party apps.

So, I wrote down a small Lucene.Net based file indexer and query program. You index by
mono LuceneLocate.exe /path/to/index/dir index /directory/to/index
and query by
mono LuceneLocate.exe /path/to/index/dir query query_term
Pretty simple program, 85 lines of actual code. Incredibly fast performance. Using external program ('cat') to index files in a directory (recursively), it indexes 180 files in 0.06 seconds. Query returning 44 results took 0.0015 seconds . Takes 24 MB virtual, 5.3 MB RSS-Shared. No GUI yet. I could have added a scheduler to pause for 10 seconds after every 10 files (5 more lines). This Lucene.Net based crawler and indexer beats beagle in performance but nowhere close to beagle.

Maybe beagle is not a lucene-powered locate. After all, to err is human.

And PLTC means ...

I just now saw the programmer's personality test webpage in Digg and decided to give it a curious shot. And ... my programmer personality type is PLTC.

(Update: The link to my personality doesn't work. I thought they have one static page for each possible combination. I am probably wrong, let me add the full text here)

You're a Planner.
You may be slow, but you'll usually find the best solution. If something's worth doing, it's worth doing right.

You like coding at a Low level.
You're from the old school of programming and believe that you should have an intimate relationship with the computer. You don't mind juggling registers around and spending hours getting a 5% performance increase in an algorithm.

You work best in a Team.
A good group is better than the sum of it's parts. The only thing better than a genius programmer is a cohesive group of genius programmers.

This one I like the best.
You are a Conservative programmer.
The less code you write, the less chance there is of it containing a bug. You write short and to the point code that gets the job done efficiently.

OMG. That's totally like me. Or whatever!

kBeagleBar is alive!

I did not guess that people still use kBeagleBar, there are packages built for it and there was even an article about it and other KDE beagle search tools. Nice!

Faceless Bugs and Advanced Users

These are really two very different topics but they came to my mind while reading about Linus' Gnome patches and bugs.

The first is about creating a new account when I need to report a bug or submit a patch for some software. Most of the projects prefer attaching to bugzilla or they send it to their member only mailing list. I am extremely reluctant to create new accounts, so I have created bugzilla and mailing list accounts for KDE and Gnome. That covers a lot of ground. But still now and then I face a need to send something to somewhere else and bam! Sign up for an account sir! There is definitely merit in this approach, since otherwise bugzilla and mailing lists would be flooded with spam. But it definitely keeps me from submitting patches or commenting on something due my lack of interest in new accounts. Last week, a friend of mine (the inventor of Sperner's Game) was trying to install Kubuntu in his brand new Lenovo T60 when he spotted some typos in the installation windows. He was ready and willing to file a bug in Kubuntu and was told to create a new account for kubuntu bugzilla. As always, he was supposed to get a confirmation email.

The email came 12 hours later and I do not know if the bug was ever filed! Even if the email was prompt, the desire to report a bug has to be high enough to cross these technical potential barriers. *sigh*

This week I made extensive addition to beagle query syntax. There is an open bug in bugzilla asking for a visual way to add these advanced query expressions in beagle-search. I was thinking how best to achieve that; it is not easy to capture the power of beagle query expressions in a gui. I found the answer while reading some posts in desktop-architect mailing list about Linus' patch. There is nothing like an expert user or a novice user. Users always try to act as if they are smart and take the path of the expert user. Presenting different set of options for these different class of users does not work in practice.

In a similar style, there is no need for a GUI for advanced query expressions. Novice users i.e. users who will simply enter search terms will never know what a full boolean query expression does (with those OR and excluded expressions). On the other hand, expert users who know how to deal with the boolean expressions, the different keywords to do property search and other advanced syntax can anyway write it by hand. In fact, it is much easier for them to write it by hand than to do it visually. In this matter, I like the approach taken by
Google. I think I will push towards a simpler advanced search UI for beagle-search and Kerry, some simple choices like choosing type of file, extension, date range etc. Write the query by hand if you need that extra ounce.

beagle memory usage

Setup: Fresh run of beagled with only the kmail backend. IndexInfo report about 13700 items i.e. mails and indexed attachments. Beagle version is post 0.2.16, so that includes the individual items in the archive attachments as well. I started beagled as exercise_the_dog, indexing finished within an hour and this is the state after indexing is over.

VIRT      RES   SHR     COMMAND
--------+------+------+--------------------
167m    55m  11m     mozilla-firefox
248m    29m  2856    X
137m    20m  15m     amarokapp
72812   19m  6884    beagled-helper
89560   18m  13m     kmail
35816   15m  11m     konsole
42780   15m  14m     konqueror
49088   12m  5860    beagled
40320   11m  9524    kdesktop
42004   11m  9.9m    basket
32620   10m  9588    kmix
43896   9252 6220    kicker
37904   5884 3908    kded
34560   5600 2680    net_applet

Remember the rule: an approximate idea of the memory usage is given by RES-SHR.

beagle:Eat less, talk less be smart

Yesterday, beagle 0.2.16 was released. A couple of weeks back, we released 0.2.15 but I did not write about it. 0.2.15 came with a lot of performance and memory improvements, new backends, new features, lots of important changes . In the process, it also broke a few things. Those were fixed and 0.2.16 is a purely bugfix release for 0.2.15. I am considering 0.2.16 the best ever beagle release. Incidentally, 0.2.13+ releases somehow or the other had some nasty problems.

Combining 0.2.15 and 0.2.16, these are the major improvements:

* Very important, the looping bug is fixed. I would even like to claim, fixed forever. I happened to find some important clue while scanning the logs and other information provided by some of our very friendly and helpful users. Eventually our 3 year old database schema was found to be incorrect. Joe finally cleared the mess. Thanks Brian and Rick! This also means an end to the "log file filling hard disk" or "beagle indexing even after a week" type problems.

* Beagle uses some external tools to filter files e.g. pdfinfo, pdftotext, mplayer yada yada. These programs are well written and almost always work. Except some very malformed or wrongly detected mimetype file is sent to them and they go berseck taking up insane amout of memory or CPU time. Since the early release, we used to maintain that there is no way we can control the external processes. After all, we just use 'em. Joe finally put an end to that excuse by using some smart rlimit tricks to limit the resources used by these external processes. We still cannot control how mplayer might behave if given an word doc file, but if it behaves badly it will be killed before too long.

* Indexing data is a strenuous job. Think about all those heavy applications which process or generate these files. But people want indexing to be as silent as possible. There are frequently recommendations that beagle should use high nice, low system priority. low IO priority etc means to be as unobtrusive as possible. The fact is, beagle already does that. However, now we even go one more step by using SCHED_BATCH scheduler policy.

There are other side improvements too, RTF filter is new. The current one is based on the legendary RTF parser by Paul Dubois. Image filters are almost new; we now have Konversation (KDE IRC client) and KOrganizer (KDE tasks and eve nts scheduler) backends. By the way, soon after 0.2.16 was released, Opera webhistory backend was added to trunk. You can just drop the binary from here into your 0.2.16 /usr/lib/beagle/Backends folder and start using it, err... trying it. I do not know how complete it is.

I would like to end by thanking the excellent user base that beagle has developed. Without them, it would not be possible to fix a whole lot of these problems. Beagle would not be what it is today without them.

Seekable LineReader

Recently I need a way in StreamReader to get the position of a line which can be stored and later skip to that line directly. C# StreamReader API does not have any way of doing this, except calling Read() repeatedly and then doing the processing yourself. Which is clumsy. Note that, StreamReader.BaseStream.Position might be wrong due to underlying bufferring. I thought this should be a fairly common requirement and indeed, many people have same question on google groups or other forums. No good answer though. One reason is, such a thing does not really make sense for arbitrary Streams, since it might not be possible to seek in them.

I needed such a thing desparately, so I created an interface:

namespace System.IO {

 // A linereader interface
 public interface LineReader {

   // Returns a position marker, which can be used to navigate the lines.
   // Some implementations may only allow moving in the forward direction.
   // Might be different from line number or file offset.
   // Should only be used for traversal.
   long Position {
     get;
     set;
   }

   // Reads and returns the next line, null if EOF
   string ReadLine ();

   // Reads the next line and returns a stringbuilder containing the line
   // The StringBuilder returned could be the same one used while reading,
   // so it should not be modified and its content might change when readline
   // is next called.
   // This is the most worst horriblest API I ever designed, for sake of speed
   // And thats why this should not be a public API.
   StringBuilder ReadLineAsStringBuilder ();

   // Skips the next line, return true if successful
   bool SkipLine ();

   // Skips required number of lines; returns actual number of lines skipped
   long SkipLines (long n);

   // Close the reader
   void Close ();
 }
}

Beagle source contains the interface and several implementations.

Fasten your seatbelts; we are ready to ship

There was a discussion going on in beagle mailing list sometime ago where I made a comment that I dont think beagle is newbie ready i.e. plug-and-play yet. Beagle fans did not understand my comment and people replied why they think I am wrong. I would be glad to be proved wrong; but all of their arguments were how they were using beagle since version x.y, how beagle was shipped and enabled by default since last z releases of some distribution and how someone is able to install beagle in a large enterprise. Duh! None of these prove that beagle is newbie ready. All they do is show that beagle works and even I know that. 0/100.

I have the feeling that some of the beagle devs and followers live in the garden of Eden surrounded by a high wall of reality. Sometimes they should go out in the streets, check the bugzilla of other distros, go through user blogs (which mostly contain complaints about how beagle does not work and how to disable it), and visit some user forums where a lot of questions are how to disable beagle from starting at startup. These are laborious jobs and not pleasing. A lot of them contain flames and invalid reasons. But almost always they are started by someone who found beagle causing trouble.

Here are some links which can make your task easy:

• Ubuntu beagle bugs
• OpenSUSE beagle bugs
• Fedora Core beagle bugs
• update Gentoo beagle bugs
  

I sometimes make the rounds and all I see are I make a point of uninstalling beagle on all my machines and The first thing I did after ... was to uninstall beagle and now my machine is happy. Silly men, how can they not like the doggy!

Subversion arrives. Finally!

KDE uses SVN for their source code management. GNOME used to use CVS till yesterday, which means beagle too was managed with CVS. I do not know the technical details. but time a again we did face technical problems with CVS. I was mostly told that life would be easier with SVN and lo behold! GNOME has switched to Subversion as its SCM (actually, still switching).

The last time this was tried by the awesome GNOME guys, they later found a glitch and had to cancel the migration. As a result I lost a commit that I made within hours of SVN migration. This time I will play safe and watch it for a few days before committing anything. If everything works out, life should be easier. Joe already cleaned up quite a bit of the unused files and directories, renamed the Evo-mail backend correctly and updated the links et al. A New Year with a clean, new repo. Sweet.

PS: There is one downside though. Joe (and others too) would like to use the SVN commit messages for creating Changelog files during creating a tarball. Which basically means others cannot observe the Changelog file between releases to figure out what was changed (neither I nor Joe updated the Changelog while committing, so this is a lame excuse). The real trouble is now I cannot write any lame jokes in my commit messages. Life would be serious now. Boo hoo.

My time with the doggie

Today I read about the Ohloh project (http://ohloh.net) and added my favourite project beagle (http://beagle-project.org) to it. I was curious what it actually does.

It took them nearly 4 hours to download and analyze the source code. But it was worth the wait. It showed some interesting statistics, like 122,885 LOC codebase, 82 direct contributors (committing in CVS) and 13 of them in last 12 months.

Just for a light comparison, Firefox has a codebase of 157,207 LOC, Amarok has 169,288 LOC and (take this) PHP 6.0 has 599,805 LOC.

It was also amusing to see my share in the project: http://ohloh.net/projects/3826/contributors/21154