FOSS meets ENG

Techkriti brings back fond memories. Techkriti is the annual technology fest at IIT-Kanpur (y'no the place where they figured out how to decide if a number is prime in a theoretically fast way).

That was one fabulous event I actively took part in during my undergrad days. A perfect mix of all kinds of technology. I was there in its early days of infancy; if I remember correctly the first prize for the software contest in my first year there went to a graphical calculator program in Tcl/Tk. By the time I left it was hugely popular and there participants from all over India. I was one of the organizers for Tech Olympiad in my final year. It was equally fun to come up with challenging problems where various concepts tie in together. The participants loved it.

This year Techkriti is even more exciting. I read in the news and blogs about how FOSS is catching up in the subcontinent. This time they are organizing a FOSS event for Techkriti, probably for the first time. I am hoping the event becomes a success, though the schedule page is a bit empty now. Beagle Xesam adapter author Arun and Web interface dude Nirbheek are among its organizers, so I am sure I will get first hand information about all the exciting things that will happen (how about a hackfest for Dashboard ;-).
.

I will end with two simple brain-stormers.

The first one is of mathematical type, my favourite: Show that any 5 consequtive numbers will always contain some number which is prime to the other 4 numbers.

The second one relates to programming (somewhat), my hobby. You are given two arrays A1 of size m+n and A2 of size n where only the first m slots of A1 are filled with sorted integers (increasing order) and all the slots of A2 are filed with sorted integers (increasing order). You goal is to merge the two arrays into A1, in linear time (no block array copying or other tricks) but (here is the twist) using no extra space (i.e. no placeholder variable to hold temporary values).

Good luck!

Open letter to OpenSUSE users

(long post warning)

Dear OpenSUSE users,

Recently I came across several threads in various OpenSUSE mailing lists
[1], [2], [3]. I was both amused and felt sorry while reading the posts. No
really some of you write funny emails. That aside, people, especially those
using FOSS don't make up things like this. I am sure the problems that you
faced exist (or existed in whatever version you were using).

I joined the project later but I still feel responsible for the sleepless
nights some of you have had due to beagle, trying to imagine what you would
see beagle has done to your computer when you wake up. I would have felt the
same if I were in your position, in fact I sometimes feel the same for one of
the browsers that I use.

There were lots of suggestions and speculations. There were suggestions of
filing bugs with us. While I do appreciate if some you can file bug reports,
I sympathise with those who dont want to open yet another account to file
bugs or email the mailing list. I belong to the latter group, so instead of
replying to the thread, let me take a minute here explaining how we try to be
friendly to your computer hard-disk space, memory and CPU.

* We nice the process and (try to) lower the iopriority.

* Extracting text from the binary files, without rewriting the app which
deals with files of that type, is an expensive operation. So, we index them a
few at a time with sufficiently long wait in between. The wait period is
longer if the system load average is high. But if you are playing games or
doing other CPU intensive operation, you will not miss the CPU spikes. Normal
uses should not be hampered though.

* During crawling (for files, emails or browser cache) we try not to disturb
the existing vm buffer cache.

* We believe once the initial indexing is over there should not be noticable
effect from beagle, so we crawl a bit faster when the screensaver is on. But
we provide options for you to turn it off.

* We use a separate process to actually do the dirty job of reading the
files and extracting data. As a failsafe measure, if the memory usage of that
helper process increase too much we kill it and start a new helper process. I
would like to claim that for the last several versions I did not see/hear the
helper process being killed due to memory problems.

* For certain files we need to store indexable data in a temporary file. We
make sure we delete it as soon as the indexing is over. There were problems
in some old versions where the files would not be deleted (they definitely
wont be deleted if you kill-9 the process) but I have not heard about this
problem in recent times.

* To show you snippets of your searched words, we store the text data of
some types of files (not the text or c,c++ kind of files whose text data is
basically the file itself but the files in a binary format). We try to be
smart here to not create thousands of small files on the disk (I have about
20K mails generating at least 10K snippet files). In addition to it, we
provide ways for you to turn off the snippets completely.

We do care for your experience and certainly for my own experience while
indexing my data. So where do we go wrong:

* Once in a while the indexer encounters a file for which it ends up in an
infinite loop. Most of the times it is generally a malformed file but
sometimes it is also our fault.

* C# has lots of advantages and one of them is that the developer does not
have to worry about freeing memory after it is used. Depending on someone
else (in this case the garbage collector which frees the memory for us) has
its pros and cons. But one thing for sure (assuming mono is not making any
mistake in freeing), there is not going to be any memory leak of the kind we
are afraid of in C or C++. Neither are we afraid of segmentation faults due
to memory corruption. If you are wondering how do some of you see beagle's
memory growing, let me remind you that "to err is human". With sophisticated
tools to prevent simple errors, comes sophisticated errors. A simple example
could be like storing in a list all files beagle finds during crawling, but
forgetting to remove them once the data is written to the index. No, we never
did that but sometimes we make similar mistakes.

* We would be extremely happy if beagle only used C# for all its operations.
Unfortunately, we have to depend on a lot of C libraries for indexing certain
files. Sometimes memory leaks (the C type) and segmentation faults happen in
them. These are harder to spot since mono does not know about the memory
allocated in the C libraries.

* Beagle re-indexes a file as soon as possible once it is saved. It is in
general not possible to know whether it was the user pressing ctrl-s in
KOffice or a torrent app saving the file after downloading one chunk of data.
As a result, beagle performs horribly, yes horribly, if it encounters a file
that is being downloaded by a p2p/torrent app. You are bound to see almost
continuous indexing as beagle strives to index the updated file for you in
real time. Same goes for any large, active mbox file in the home directory
_not_ used by Thunderbird, Evolution or KMail (for mbox files of these apps,
the corresponding backend is smart to index only the changed data).

* NFS shares have their own share of problems with file locking, speed of
data access etc. We have tried to deal with them in the past by copying the
beagle data directory to a local partition, performing indexing and then
copying back the data directory to the users home directory. It is a feature
not continuously tested and I am sure you can think about lots of cases where
this would fail.

* The first attempt to write a Thunderbird backend was a disaster. Well, it
was good learning experience for us but it will cause headache to most users.
We disabled it in the later 0.2 versions. There is a new one in the 0.3
series which reportedly works better.

* There was one design decision which backfired on us. Imagine you dont have
inotify and have a large home directory. To present you changes in real time,
one option is to crawl the directories regularly (kind of what the WinXP
indexer does). You can imagine the rest. Though inotify is present in the
kernel these days, the number of default inotify watches (the number of
directories that can be watched) is pretty low for users with a non-trivial
sized home directory. In the recent versions, we disable the regular
recrawling.

* Besides continuous CPU usage and hard disk activity (for days and weeks
after the initial indexing is over) the above had another effect on the log
file. Add to it the pretty verbose exceptions beagle logs. We want to know
about the errors so we still print verbose exceptions but we dont reprint the
same errors anymore. (I have been told that some of the OpenSUSE packages
have the loglevel reduced to error-only which will automatically generate
smaller log files).

* This is a good excuse to end the list with. C# and beagle architecture
allows us to add lots of goodies. After all, we (read I) work on beagle
solely because I love to play with it. The more the features, more the lines
of code and more errors. The only good part is once spotted, they are easy to
fix. Check our mailing list and wiki for the available freebies.

So in summary, we try to be nice to your computer (and to you ? maybe, if you
are nice ;-) ... just kidding) but there are limitations that we are
constantly trying to improve on. Any of you can look in our bugzilla, our
mailing list archive, our wiki or hang out in our IRC channel to see for
yourself how we try to issue any problem with utmost importance. Ok, I lied,
as much as our time permits. There are lots of features in beagle and some of
them rarely get regular testing, mostly because none of us use those
features. I wont be surprised if there are major problem with these. I assure
you that if you bring any problem to our notice, they will be taken care of,
if not completely resolved.

Lastly, I read in one of the forum posts that beagle-0.3 will land in factory
sometime soon. If any of you wants to verify the facts above (the good ones
or the bad ones ;-), give that a spin. And a friendly suggestion, if you only
want to search for files with certain name or extension, you can do much much
better with find/locate.

Your friendly beagle developer,
- dBera

[1] http://lists4.suse.de/opensuse-factory/2008-01/msg00157.html
[2] http://lists.opensuse.org/opensuse/2007-12/msg01796.html
[3] http://lists.opensuse.org/opensuse/2008-01/msg01083.html (could not find
the parent of this thread)

klik2 klik beagle

Yay! I managed to make a klik2 recipe for beagle. In principle this should enable anyone to just do
$ klik klik2://beagle
and happily run beagle. All the dependency packages will be automatically downloaded and managed in the background. Or to download once and use many times, you can do
$ klik get beagle
$ alias runbeagle='klik run ~/Desktop/beagle.cmg'
$ runbeagle

Not all of the above is happening right away; klik2 is under development and is looking promising but not completely done yet. But if you already have mono and want to run beagle, you can sort of do it now. This works for any distribution that klik2 works for, which is basically almost all the major distributions.

1. Get and install klik2
2. Download http://cs-people.bu.edu/dbera/blogdata/beagle.xml
3. $ klik get beagle.xml
• It will do a lot of stuff and end up failing since there is no single app called "beagle" in the beagle package. It will however create the file ~/Desktop/beagle_0.3.1-2.cmg
4. Run beagled.
• klik run beagled ~/Desktop/beagle_0.3.1-2.cmg --fg --backend manpages --backend Files
5. Browse to http://localhost:4000/ to access beagle using the web interface. You can use it to search, check indexing status and shutdown beagled.
6. To run the other beagle tools, the pattern is the same
• $ klik run beagle-command ~/Desktop/beagle_0.3.1-2.cmg beagle-command-params
• For example, to shutdown from command line
• $ klik run beagle-shutdown ~/Desktop/beagle_0.3.1-2.cmg
7. The next time you want to run beagled, you need not run the recipe again; start from step-4 straight away!
   

one zero-three-two sailed today

Beagle 0.3.2 was released today. On one hand we are still catching up with the regressions and new bugs that were introduced in the mighty beagle-0.3.0 and on the other hand, new features are streaming in. While yelp remains broken with beagle, I was amazed at how easily I can search within the manpages and double click on the results to open them in yelp. A much better alternative to man -K.

In other news, Lukas has started working on providing spelling suggestions Did you mean ... ? There are some technical limitations which are not fully resolved yet so it did not make it into 0.3.2 . It is currently housed in a branch and I hope to release it into the wild soon.

Beagle was not designed as an RDF store at its inception. It will take quite some work to make it a genuine RDF store. But what if there was an RDF adapter that sat between an RDF client and beagle and talked to each of them in their corresponding language, yet maintaining sanity. There is an ongoing work to overlay a Semweb selectable source on top on beagle. We will see how that goes.

A post on klik2 rekindled my desire to create a klik package for beagle. It will be easier this time since klik2 handles command line programs. I tried the automatic debian generated recipe and it mostly worked. Mostly, because one of the tools in beagle ran with --version and --help but failed to find some libraries to do anything more. I think all I need to do is to teach klik how to set certain PATHs and environment variables. Pretty exciting, what do you think ?

Enterprise search OR How to index on-demand

If you are like me who keeps their filesystem organized, have a relatively
unchanging home directory or just simply do not want realtime indexing, you
can use beagle-build-index to meet your needs.

Beagle-build-index builds static indexes from files. Static indexes are
created and updated on demand when beagle-build-index is run but the
directories are otherwise not monitored for changes. The next time the
command is run, the changes are registered in the index. Once the static
index is created, you can ask beagled to search in it (by
passing --add-static-backend /location/of/static/index). beagled need not be
stopped while running beagle-build-index, it will automatically use the
updated index for searching once beagle-build-index finishes.

That was for files. If you want to do the same for anything else, say emails
or notes or addressbook and you do not want realtime monitoring, start
beagled normally and let the indexing finish. Then stop beagled and restart
with the parameter --disable-scheduler. Unfortunately, to update the index
with changes, beagled needs to be stopped, started normally and allowed to
run till updating of index is done, stopped and then again started with that
parameter.

If you are a system administrator managing lots of users and you dont want to
run beagled in realtime indexing mode for all of them, you can use the above
procedure to create/update static indexes, say once a day.

Or if you don't like mono, you can use Recoll for files and mairix for emails.
There are probably many more such tools but these are the two I know. Just in
case you have not heard about Mono, it is an open source implementation of
ECMA standard compliant C# compiler and a Common Language Runtime. And some
more goodies, all in all pretty useful.

Many reasons to like, what's yours ?

Beagle 0.3.0 was released beginning of December. It is nearly 2 years since 0.2.0, more than 10 months since the last feature release and it has been about 2 weeks since then. In the mean time we identified some problems upgrade problems with 0.3.0 and released 0.3.1 and Mono released 1.2.6.

In contrast to 0.1.0 and 0.2.0, beagle-0.3.0 did not have any single major-impact change. But there were lots of small changes, all over the summer months and the months following them. It was getting increasingly difficult to handle all the small changes without going through the "Release early" trick and at some point we paused development, did a test release and then finally released what we have as a major release. I am personally expecting a fair share of bugs and regressions.

What are these small changes anyway ? I will leave out the invisible ones, some of which I have blogged about before, and only explain the ones that will directly impact your desktop usage.

There are 3 new backends: the Thunderbird backend (newly written, much better than the earlier one), the Opera history backend and the Nautilus metadata backend. There is also the TeX filter, one of our most demanded ones and new audio filter based on Taglib-sharp. There are new Firefox and Epiphany extensions which do a lot more than indexing browsing history and bookmarks.

The UI got some love; specially a bunch of useful options were added to beagle-settings like the backend selecion list. For obvious reasons, users should disable the backends they are never going to use.

One of the side effects of the beagle textcache previously was the creation of thousands of small cache files on the disk. People reported that the external fragmentation was wasting a lot of space. The textcache module was redesigned to minimize the fragmentation; I am sure you will appreciate the recovered space. We also compacted the external attributes; besides other benefits that will save some more space.

Two major enhancements were made to the query syntax, which is already quite rich. Date queries are now possible; date queries do not make complete sense without date range query, so that too is possible. And a new "filetype:" keyword was added e.g. to search for images use "filetype:image", to search among documents use "filetype:document" etc.

The major complains against beagle are constantly high CPU load, high memory usage and improper termination (or not exiting at all). The first two are well known and oft discussed. The third problem is not directly brought up, but have been found to be the reason upon close investigation. I gained valuable experience trying to find my way through the web of signals, threads and events in beagle code; a number of key issues were spotted and fixed. Oh, and the first two issues were also dealth with, as much as we could diagnose, but that is nothing new. It will sound funny, but a few of the high CPU and memory problems are direct results of some of our decisions that backfired. Some of them were fixed and the others being worked on.

2 experimental features were also added. One is a web interface to search beagle from Firefox (gecko based browsers really). You can also create standard bookmarks for common search terms. The neat thing about this web interface unlike the earlier webservices based one is that there is no heavy weight server running on beagle's side. This one communicates with beagled using BeagleClient XML based API and builds the entire GUI on the client side; a pure Web 2.0 AJAX/XSLT/CSS webapp (ok, these are some cheap buzzwords).

The other fancy feature is searching other beagle daemons over the network. Using Avahi you can even publish your beagle daemon or discover other beagle daemons in the network. We haven't quite figured out how to handle security, authenication and some other issues. So the feature is disabled by default and marked as experimental but I believe it can be used in some innovative way.

We received request from some distributions about global config files; useful for both distributions and sysadmins. Some useful global configuration settings would be to exclude certain directory from indexing for all users, adding or removing file ignore patterns from the default list, disabling of KDE backends by default in pure Gnome distributions. Some of the options were moved from the code to the config files, so that they can be set globally and overriden by individual users.

These are only some of the major ones.

Lastly, the reason I got excited about mono-1.2.6 is because it has some fixes and improvements that will be directly visible when using beagle.

Hide and Seek where I lost!

Once upon a time someone created a Bittorrent website named SeedPeer [1]. Over
the years they grew bigger and popular. Then they also wanted to write a
bittorrent client. Recently they announced that their client (named SeedPeer
too) is mostly ready.

The above part is known. Then, Alan, the author of very-much-open (MIT/X11
licensed) MonoTorrent posted his objection that SeedPeer was based on
MonoTorrent (is it a rip off ? or based on ?) [2]. What annoyed him was that
the SeedPeer folks said nothing about MonoTorrent anywhere and claimed that
the code is all theirs. Quite right for Alan to be disappointed. I read his
blog, chuckled and said to myself ..."sigh! yet another victim of MIT/X11 and
BSD style licenses"[3] (coincidentally, I have read several posts in the last
few weeks that somehow end up with the openness of the above licenses; BTW,
beagle is also under MIT, which means you can sell it too...).

I like playing hide and seek so I downloaded SeedPeer to see how they managed
to agree to MIT/X11 and not say anything publicly. Quite possibly they credit
the author in a sneaky way somewhere. Ahh no. The exe and other related files
mentioned nothing of that sort. Ok, I went ahead and posted a comment
pointing out that SeedPeer people need to abide by the license [5].

For some reason I had that page open, so when I checked later a lot of people
were complaining of plagiarism and stuff. Probably someone even claimed that
SeedPeer has nothing do with MonoTorrent (it was something like "just because
its in C# does not mean ..."). I said to myself valid point, maybe Alan is
wrong... and thought to perform a silly test: compare the output of "strings"
of SeedPeer.exe [6] and MonoTorrent.{exe,dll}. Hmm... there was a lot of
matches, not only coincidental ones but suspicious ones. Like SeedPeer.exe
had "Mono.Math" in it - why was SeedPeer using Mono if they claimed before
that they are only focussing on Windows for the time being. Then there were
lots of strings that I only found in MT source and with uncommon names. I
posted a message in jest, saying I found a huge list of common strings and
that they probably have a few minutes to mangle the class/method names in
their source.

It was quickly followed by another post (from one of the admins, I suppose)
that they are not afraid and I should post the list. Also once their software
is complete, they would open source it. That kind of ticked me off ... could
I be that wrong ? Could the common words in "strings" be that coincidental ?
Well, off to "monop2 -p", the easiest "decompiler" at hand. Got the names,
removed the namespace and did a direct comparison. Irrelevant, but I could
not find any one line shell command to return the common lines from two
files; google helped me find a one line perl command. Anyway, there were 109
matches! And a lof of them were from private classes in MonoTorrent. The
similarity was too striking to be a coincidence. I modestly reported my
findings.

I have not been able to access the site since then. It was down; I should say,
it was down from my computer. Because, curious, I accessed the site from
another computer and it was nice and jolly up. Except ... the SeedPeer.exe
article, the download link to the exe and some other news articles are gone
[5]. I guess they "banned" me for, I guess reading their policy on their faq
page, "using comments section to insult others" (that was the closest match;
I also could not figure out if they mean blocking IP when they say "banned" -
since I am not a member at all).

I wished I had accessed that page using Firefox. Beagle firefox extension
would have indexed it right away and I would have a copy of the forums page.
I used Konqueror which does not always save the pages in its cache directory;
I just checked, there is no cache file for the forums page.

Like Adleman [7] (ahem), I like to break stuff. And thats what I did when they
argued that SeedPeer is their own and solely theirs. But I suck at
hide-and-seek, I really do!

[1] http://www.seedpeer.com
[2]

http://monotorrent.blogspot.com/2007/11/i-heard-some-great-news-earlier-in-week.html
[3] http://www.opensource.org/licenses/mit-license.html
[4] http://beagle-project.org
[5] http://seedpeer.com/sitenews/article/31.html
[6] http://cs-people.bu.edu/dbera/blogdata/SeedPeer.zip
[7] http://www-groups.dcs.st-and.ac.uk/~history/Biographies/Adleman.html

Good Luck Joe

Please join me in welcoming Joe Shaw, the second non-Novell maintainer of the Beagle project. Joe has solid knowledge of the codebase yada yada yada...

KGiMAP

A not so beautiful Allston morning turned out to be a rather useful one with the slashdot post [ Free IMAP On Gmail]. I checked my gmail settings and (apprently I am among the fortunate ones here) I saw the "Enable IMAP" setting there. Wee...

Fired up Kmail, added a new account for Google as a disconnected IMAP. And on my way to sync ... I was curious how the labels were going to be handled, I have about 15 of them based on which mailing list or which bugzilla an email is from (I have mentioned this before - I prefer organization over search). It was nice to see the labels getting downloaded as imap folders. And there is a special folder [Gmail] which contains the regular gmail folders - All mail, Starred, Trash, Spam, Sent Mail. Thats when it struck me!

All the emails are going to be duplicated or triplicated or could have even more copies. Labels are not disjoint, so there are mails in my Inbox that are also in beagle-bugs and of course all mail ever is also in All Mail. To add to the trouble, deleting an email locally is going to archive the email (basically, it removes the Inbox label from the email) and not delete it. The huge amount of mails in the Spam folder is not healthy for my hard disk space. The huge amount of emails my beagle-bugzilla filter sends out to Trash is similarly bad.

As of now I resolved the situation by asking all my filters (I have one filter for each label) to archive the email ( Skip the inbox) and then apply the label. I asked KMail to not subscribe to any of the [Gmail] folders. And I created a Gmail label(IMAP folder on KMail) Totrash which I marked as my Trash folder in KMail (for the GMail account). Now whenever I delete a Gmail email from KMail, it will go to Totrash which I can empty periodically from the web-interface. What a kludge! I wish there was a rule to empty folders or a filter move emails from Totrash to Trash!

There is no third way

(This post is not about beagle though it is hosted on PlanetBeagle.
However it is somewhat related to "search".)

I have seen two approaches to finding information, in land of software
as well as real-ware. There are people who find it themselves and
there are some who use an external agent to get what they need. Human
memory is expensive, so these are the most obvious approaches. I have
seen equal number of people on both sides.

In the first group are people who like to keep everything organized,
from physical files to computer files, books to RSS feeds, knives to
applications. Organization typically involves grouping the items into
categories and re-grouping again if the number of items in a category
grows too much. Its a pain to remember things, then there is so much
to remember. Care is taken so that the number of categories don't grow
too large or the nesting isn't too deep. Computer Science people
cashed in on such problems, analyzed them to death and presented us
with databases, shallow and deep trees, hashtables, sorting etc.
Whatever the approach, the objective remains that reaching one
particular item should take as less amount of memory and labour as
possible. Memory and labour appear as trade-off here, so there are
variations of this where the most frequently accessed items are
optimized more than others (compression, source coding, amortized
algorithms, cabinets with upper and lower shelves etc.).

The second group of people use an external agent to do this task. In
reality, not everyone belongs to only one group, people tend use
whichever approach is suitable. Anyway, the external agents could be
one's mom, secretary :), a database, a filesystem, search engine etc.
They are nothing but efficient members of the first category. The
efficiency matters here; tolerance gets extremely low if I am not
doing something but hiring someone to do it.

It is clear that when dealing with items on the start menu, the same
ideas would come into action. OK, so the rest of the post is about
start menu, you can stop reading now.

Either we have a menu, properly organized or we have a dump of
everything with an extremely efficient way to get what we want. OK, we
can do the optimized variation of the first and add search to it to
make it work for the second group (e.g. kickoff, internet with search
engine and directories, filesystem with desktop search). But to get it
done right, both should be done with precision and care. Which means,
the search has to be fast and accurate and categorization should not
be too deep nor too flat. Geeshh... that is difficult.

Actually not that difficult for applications. If done the right way,
they can be arranged in a start menu with categories (not Windows
style though!). Or they can just be there and you find them using a
search tool (quicksilver, katapult, deskbar). Or both (kickoff,
kickoff spinoffs).

This is where I found Mandriva to go wrong. They had a nice start menu
to begin with (because Mandrake had a nice start menu to begin with).
In the latest distribution, the Mandriva specific categorization has
been replaced with a two level (i.e. top-level with
network/tools/office and then apps in them) menu. Some of the submenus
became too large, and they have been put behind another nesting called
"More" (e.g. "Firefox" went behind more because it not native KDE).
The end result is a menu, which is only two level and pretty flat (so
less to remember which submenu to go to) but once in the submenu, have
to memorize the position in a pretty long vertical list. Add to that
the effort in dragging the mouse a significant vertical distance. I
have entries that requires me to drag the mouse all the way from top
(I have my start menu button at the top-right corner) to the bottom -
that's 1024 pixels. The earlier menu was the same with one more
nesting, like "Browsers", "Spreadsheet", "Terminals" where I only had
to memorize the levels and then look in the 2-3 items in each
immediate category. The vertical distance covered in the current menu
is larger than the horizontal distance due to that extra submenu.
Adding to the woes is the lack of an embedded search option.

There is another aspect with menu which make vertical movement much
harder than horizontal movement. Its related to Fitts's Law. But first
a small digression. I like to call it Fitts's Theory. Because it
describes certain operation in a particular model. Calling it law
isn't wrong, except it makes it sound like anyone not following the
"law" is from Mars. Once again, it tries to model some observed
phenomenon, and does not aim to dictate how the phenomenon should take
place. Unfortunately most commentaries about Fitts's law seem to think
we should all abide by Fitts's law or leave this planet.

Anyway, in menus the entries are horizontally layered. The menus are
designed to open a submenu when the mouse in on the corresponding
entry. The problem happens if I try to follow a straight line from an
entry to an entry in a submenu. If the entry in the submenu appears
significantly lower in the submenu, the mouse will inevitably cross
over the next entry in the menu, which will open the submenu for the
next menu entry thereby forcing you to start again. Vertical motion
isn't unrestricted. It happens to me always when I quickly try to go
from the start menu to some application whose position I know thereby
trying to go straight from the button to that entry. If I am not
careful, I end up opening the submenu of the next menu entry in a
haste, then move left and up to find the right menu entry, then move
right _carefully_ to horizontally enter the correct submenu and then
rush down to the memorized spot. All in a wink of an eye and it has
become my second nature now. But every time this happens, I feel
slightly irritated. If menus are designed to take advantage of the
horizontal freedom of movement more, it would be much smooth. Deeper
nesting has the mentioned side effect. By the way, radial menu or
submenus that open in the middle or in the first entry should be
better. But apart from a few radial Firefox context menus, I have not
found any other implementation. I love radial menu.

That is all I had to say about start menus. Before finishing, I should
point out that even for the first group of self-organized users,
having someone or something to search is not totally out of the
question. Some form of data are too vast to organize (bookmarks and
directories wrt internet), some have no single way of organizing or
some even have no of meaningful grouping criterion. And even if you
arrange all your computer files in directories, emails in mail
folders, feeds in feed reader folders, websites in bookmark folders,
if you are to find this blog (containing "Fitts's Law" but unrelated
to beagle), then good luck!

I saved 80MB

Its a common question in the database community where to store large binary
objects, commonly known as BLOBs (binary large objects). There are two
obvious options, store it in the filesystem or store it in a database. And by
twisting arms we get the third obvious approach, a mix of the two. There are
some studies in database academic community on the tradeoff.

Thinking carefully, the database and filesystem work in a similar way when
storing objects. Both store data in tree fashion, providing fast access to
any object. Both can add or delete object. And both suffers from
fragmentation.

The fragmentation is the key issue in my test case. So what is my test case.
It consists of lots and lots of small files and I mean, really small files
and some very large files. Just to give an example, I have a total of 20000
files. Of them about,
500 are above 4K
1000 2-4K
2000 1-2K
16000 less than 1K.

There is a reason for this extremely skewed distribution. The files are
textcaches in my home directory. The main contributing factors are about 8000
files (a mix of source code, tex, pdf and html and oh ... lots of pictures)
and 19000 emails (without spam). Now source codes are self caches i.e.
separate textcache is not stored for these files. Same with txt. Pictures and
other media has no text to cache. So the bulk of cached text comes from html,
pdf and office documents. Most of such documents in my home directory are
generally pretty large i.e. with a significant amount of textual content.
This is in sharp contrast to the behaviour for emails. The textual content
from the emails are stored. Also, unlike the html files on the disk, the HTML
parts in these emails are small, really small. Those contribute to the huge
number of extremely small files. If you are wondering how come I got 3000
absurdly small 20 byte files, I store the gzipped copy of the extracted text
in the textcache.

So now the question is how to best store data with such distribution. The
simplest thing is to use the filesystem. Filesystem are equipped with
sophisticated methods to avoid fragmentation. But with a fixed block size
(like 4K on my home directory partition), there is bound to be some
fragmentation when storing small files like above. On the other hand,
historically databases were concerned with storing small strings (with all
sorts of querying and update speedup; and they are pretty good at that).
Recently, most of the databases have added support for large binary objects.
There is also a small twist to this problem. The data is mostly static but
there are deletions (and sometimes modifications). The last important factor
is I am using the embedded sqlite database which stores all its data in a
single file; so in essence, what I get is a "fast retrieval and
update"-enabled huge tar file.

The prevailing folklore is that database is better for smaller objects and
filesystem for larger objects. What is small and what is large depends on the
nature of the problem at hand. There are some papers doing detailed analysis
of the tradeoff, and they conclude somewhat along the same lines. I found one
paper showing that adding and deleting lots of large objects over time causes
enough fragmentation in a database that it can lag behind a filesystem.

I am not much of an experimental person myself. With the above testcase, I did
not even think twice about what I should do. Well, for other reasons (like
heap fragmentation in my application, since reading from a database would
require me to store the entire data in memory) I was a bit biased towards the
filesystem approach. So I decided to take the middle path of a mix of
filesystem and database. Thanks to decent APIs, implementing the hybrid
approach did not turn out to be anything complicated. I store the first 4K in
a buffer and then if I get more data, move everything to a file otherwise
save everything in the database.

All the 19500 files with less than 4K size are now safely sleeping in the
database, which would have otherwise taken 4K each.

(* The title is not entirely correct. I did not have to waste 4K blocks for
each of 19500 files, but now my sqlite database is 26MB instead of something
much smaller. But I am confident, this is good for the mankind at large and
for the dolphins too.)

Tracker-ing looks fun

Jamie checked in thunderbird support to Tracker today. How do I know ? Because I subscribe to svn-commits-list and my gmail filter separated him out when he was adding the beagle thunderbird backend (probably with some modifications, haven not checked the changeset) to tracker.

This is looking pretty cool. I also found out Tracker-0.6 was released with some extremely useful features and a damn good UI. The UI groups the documents in an intuitive way, shows more context information from matches, allows you to do tagging inplace (and has gradients, uh-on). The core daemon (is that what it is called ?) can scan evolution mails, and pidgin logs. Now it can also do index thunderbird mails. It lists XMP metadata support too - which is nice and something I never managed to finish in beagle.

I am pretty sure people will find it useful. I could not resist myself to admire it and share it with others. Well, I was told that tracker is set to become the default desktop search engine in ubuntu, so a lot of unhappy souls are about to become happy soon.

For me, I will continue using beagle. Why ? Because, it does what I need it to do. And if it does not do something, just remember it is written in C# :). Meanwhile, you make your own decision.

How is WebBeagle for a name ?

(Or FireBeagle ?)

More...(drag mouse)

Close Tab

Forward

Next Tab

Reload-Stop

Close Tab

Back

Home Page

I always wanted to search beagle using a web-browser. My desktop searches are mostly infrequent but complicated. So if I am unable to find something I definitely need an index-based search tool (e.g. beagle) but on the other hand I don't want to keep an application open on my desktop for a long time (beagle does not consume significant memory for me, so it can keep running in the background). That was one reason why I wrote the kio slave for beagle (I have stopped liking it for a long time) and followed it with a kde deskbar like applet (which is nice but with limited functionality). Now that I am done with shameless advertisements, let me share with you yet another way to query beagle.

Today I wrote some ajaxy,xslt-ed webpages to allow users (read: me) to query beagle using a browser (err... firefox, konqueror-3.5.5 does not have XSLT processor). It uses the networkservice backend that can be used to query beagle over network (based on last year's Google Summer of Code projects).It uses internal knowledge of how queries are serialized, lists all the information in a boring way and does not show snippets (yet). But it works and was reasonably fast in displaying 42 results. Besides the boring UI, allowing browsers to access services always opens up some security hole, so it is disabled by default. If you so desire, use it but at your own risk (check the commit log for how to turn this feature on).

I do hope to get this feature properly implemented. The things need to be done are:

1. Fix the network-backend (it is sufferring from some crashes).
2. Probably related to the network backend as well, some kind of search authorization is needed.
   
3. Use CSS + javascript for the results page to group/sort the results, and make them look decent.
4. Get snippets and display them. I am thinking of retrieving snippets only on demand. Somehow the name and the location of files or the sender and the subject of emails help me more in filtering out search results than snippets. Which is quite unlike how I use web search engines.
5. Figure out a way to use the C# or libbeagle API to create the xml request messages. Currently they are hardwired. If the solution turns out to be too complicated, it might not be a bad thing to leave the format hardcoded as it is now.
6. More cosmetic, separate command line and configuration options for this feature.
7. This is more related to the network query implementation; figure out how to use the QueryDomain thing meaningfully. The results from some backends only make sense on the same machine and even for some of them, it is tricky to open the applications just from the URI itself. So does it make sense to show e.g. evolution mail hits in the browser ? Does it make sense to return gaim (pidgin) hits when queried over network ?
8. How does the browser behave (read: choke) when it receives 1000 results (100 results each from 10 backends)? (i.e. the DOMParser has to parse a huge string and form a huge DOM of 1000 Hit nodes. You see why I dont like to get all the snippets beforehand :)?
   

Hey Firefox, Beagle This ... NOW!

Consider this scenario: B was writing an article when he remembered that he read a relevant blog some weeks ago but did not bookmark it due to his lack of foresight (stupid B). What to do ? He could possibly run a grep in his Firefox cache folder hoping that the cache would still be there or scan his FF history for a familiar looking URL. Anything better ?
If B was using the FF beagle extension, he could find the blog instantly

Really! Then I will pose this requirement. B has been using the new FF beagle extension for some time now. But he does not want to index all his browsing history. Sometimes he just wants to browse off the record. Maybe after a while he wants to index the pages he visits. Pretty random ... but hey, that's B's nature.
No problem. The new extension puts a small status icon, which can be  used to disable/enable the automatic index; really, at B's own sweet will.

That was easy. How about this one, a bit harder. Someone told B about FF beagle extension, and B started using it. But this time he needs a PDF paper he found online. He does remember that he read the paper and he must have read it in the browser itself. Now what ?
If B was using the new FF beagle extension, he could even search for embedded documents. Embedded content is not indexed automatically (as of now) but can be indexed as required using the status icon menu.

Hmm... But what if B visits a page with auto-indexing disabled and then decides that he better index the current page. Now what ? Does he have to enable the auto-indexing and then reload the page. Thats stupid!
Heh. You gotta be kidding. Of course he can use the status icon menu or the page context menu to index the current page.

If B wants to index an image, or take this one, sees a link to something (PDF, image, java source file, mp3 ...) and wants to index it without loading it explicitly ?
The context menu provides options to index an image and index a link; in both the cases the file is retrieved in the background and indexed. All filetypes that beagle can filter, can be indexed using the index this link feature.

This new extension seems to uber-useful. I guess I err... B should really start using it. Can B save notes while indexing on demand i.e. if he is indexing a link he might want to index a few extra keywords (for querying or for reference) ?
Yeah! B gets that too. If B checks the corresponding preference entry, the extension will prompt for extra keywords to save and index when B explicitly asks it to index something.

I will be good and share this new beagle extension with everybody, written by Tao as part of Google Summer of Code 2007. Soon to be found in beagle trunk.

Mr Beagle, you are on the air

Back in India, I did one good thing. Finished adding Alexis Christoforides' project of Google Summer of Code 2006 Beagle over the network. Its a bit untested with the recent versions of beagle and I didn't read the code completely, so I left it in a separate branch for the time being.

What can you do with this wonderful project ?
- Start beagled on one machine
- Add that machine (with port 4000) to the list of neighbourhood nodes via beagle-config on a second machine in the network
- Remotely query the beagled on the first machine from the second machine

The SoC project was a complete one, with GUI support. However, I only added the basic infrastructure and enabled the cli tool beagle-query to query remotely. I am not sure if querying for emails remotely makes sense, how will the results be opened ? As of now, only the filesystem backend is enabled to send query over the network.

Ideally, beagle should automatically find out other networked beagled in the network. Don't despair, there was another SoC project in 2006, Avahi Integration by Kyle Ambroff doing exactly that. Sadly I am too preoccupied to work on integrating the Avahi work. But with that and the GUI parts of Network Beagle, beagle would be useful in more ways than ever.

Silence of the Dog

Lately beagle releases have slowed down quite a bit; there were a few bug-fix 0.2.16.x release and another 0.2.17 bug-fix release (it was supposed to 0.2.16.4, but the changelog was too large for a point release). The underlying goal is to get ready for 0.3.0; svn trunk is changing so rapidly these days that it is difficult to isolate the simple ones and make them into a 0.2.x release. On the other hand, the changes are too major to be put into a 0.2.x release (they would also need extensive real life testing).

Recently I moved beagle to use taglib-sharp for filtering music files from entagged-sharp. I was told entagged is no more actively maintained and taglib is definitely seeing a lot of rapid development. My timing was not quite right, the 4th March news "Entagged is unmaintained" is followed by the 28th March news "Entagged is maintained". I came to know about it only after I made the transition. Too late! On the plus side, taglib# has support for larger number of formats and is being used by Muine and Banshee, so expect sharing of taglib-sharp libraries. Unfortunately, there are no taglib-sharp packages out there yet (there is a proposal for a debian package), so all the mono apps are currently including taglib-sharp by its source. We too initially source included it, then removed it and instead linked against the package. But if there are no packages for the major distributions, it might make sense to source include it; compiling Beagle is pretty demanding anyway.

In other news, I used the extremely handy heap-shot to identify that instances of IndexReader were not being GC-ed even long after the corresponding method ended. Explicitly setting them to null immediately freed them. I suspect some thread local storage magic happening behind my back. Note to self, set IndexReaders to null immediately after they are closed. Did I say heap-shot is amazing ?!

There are several more improvements to the speed and memory performance of IndexHelper and BuildIndex. One notable feature I added was to reduce re-indexing of files which could not be filtered before. Due to the inherent distributed nature of beagle indexing, the crawler is always separated from the indexer. So if the crawler finds some file which was not filtered before, it has to re-submit it to the indexer. Who knows! There might be a suitable filter now. The downside was that a lot of files were being repeatedly re-tried by the indexer, slowing down the whole process. I decided to store the files containing the filters and their last modified times in a filterver.dat (akin to mozilla pluginreg.dat) and if the filters were not changed since last run, assume that there is no newer filter. Fair guess I would say.

Beagle knew how to index email attachments for quite some time; some months ago it also got the ability to index archives. However all along this was done by extracting the included files to a temporary file and then indexing it. This was done primarily because of the way included content (aka child indexables) were handled and also due to the fact that some of the filters only worked on physical files and not streams. This whole temporary file business never pleased me, there were race conditions which could leave undeleted temporary files in the system, even small included files had to be written to disk and further, extracting the contents of an archive to index it defeated the whole purpose of archiving it. Last week, I added the infrastructure to allow indexing of archives and email attachments without extracting them, if the filter permits of course. The infrastrusture is there, the archive and email filters should be modified to take advantage of this.

Finally, one feature I personally would like to see in 0.3 is support for XMP sidecars. XMP sidecars allow users to add a separate file.ext.xmp file containing arbitrary metadata (but in the XMP format) about file.ext. Really extensible solution for metadata. The main part of the code is in svn trunk; it still does not support renaming or deleting xmp files. Hopefully it will be finished in time.

This will probably be my last post before my annual break to the land of mangoes (" fruit of the gods"). Sadly, I have (knowingly) only tasted about a dozen varieties of mangoes, out of over 300. I will definitely try to increment the number this time. Next post, July.

Upgrade to Spring

Yesterday in a fit of mind I decided to upgrade to Mandriva latest release.
The steps included
- backing up .kde, .kderc, .qt, .gtk* and .local
- logging out of kde
- setting up a mirror as a distribution source for urpmi (mandriva is offering
non-free e.g. sun-java in its free source these days, but I still need the
PLF sources for codecs, BCI enabled freetype and fontconfig and a few other
things)
- # urpmi urpmi
- # urpmi <bunch of> kernels
- # urpmi --auto-select
and then selecting the ones I would like to upgrade from them

Soon I was running Mandriva 2007.1 Spring (Free). Yay! Its beautiful. The
Ia_ora theme and other Mandriva artwork is gorgeous. I legally own a XP cd,
from which I extensively use Verdana (for text) and Tahoma (for widgets).
They look wonderful as always with plf freetype (w/ hinting). I like to use
large fonts, enough to be readable 4 ft afar but somehow the deja or
bitstream fonts have a weird fuzziness in the curves of 's' and 'o's. I
cleared the settings of a test user account and a default new account looks
quite good (apart from the kbfx-styled mandriva menu).

KDE was upgraded to 3.5.6; I was actually using a few kde-3.5.6 packages from
cooker so there was no huge surprise. I was worried that mandriva would mess
up some of my settings when I log in for the first time as my normal user but
thankfully that did not happen. The system feels faster, konsole definitely
starts faster than before. Overall, I am extremely pleased with 2007.1; I
wish them all the best.
http://wiki.mandriva.com/en/Releases/Mandriva/2007.1/Tour

Some minor annoyances:
- mandriva kernel (based on 2.6.17) still has the weird cpufreq bug where
scaling_max_freq is same as scaling_min_freq (thus rendering all the
governors useless). It is probably the same problem described in

http://www.mail-archive.com/linux-acpi@vger.kernel.org/msg04484.html
- Suspend to RAM is broken in mm kernel (Mandriva has moved to pm-utils and it
works like a charm w/ the default kernel)
- tmb kernel has some problems of high CPU usage when copying files and broken
resume from s2ram. I still need to test the other tmb versions. I really like
the tmb kernel improvements.

Theory Guy...

This is not really techie but has a somewhat tech-vs-theory war flavor. The following list came out of today's departmental Friday lunch. Top 10 "ways to tell a theory person from a systems person":

10. One gets job and another does not.

09. One likes 10000n^2 better than n^3.

08. Space-time is important outside Star-trek.

07. In therory they are same, but in practice they are not.

06. P is not equal to NP divided by N.

05. n^10 is efficient.

04. SAT (Famous NP complete problem also known as Satisfiability) is most of the times easy to compute.

03. (Specific to our department) One can be only be found across the street.

02. Its pronounced "Lee-nux" (not "Lai-nux")

01. Computers do more than just email ?!

TinyBeagle or a Lucene Example

Recently I read this interesting comment in an OSNews article. It tried to briefly summarize what beagle is. I take users' comments very seriously and this person seem to know some internal of beagle, so I thought maybe he is true (modulo some factual errors). Maybe the only new thing in beagle is the crawler, the GUI and the scheduler; its mostly little C# glue code tying up a few third party apps.

So, I wrote down a small Lucene.Net based file indexer and query program. You index by
mono LuceneLocate.exe /path/to/index/dir index /directory/to/index
and query by
mono LuceneLocate.exe /path/to/index/dir query query_term
Pretty simple program, 85 lines of actual code. Incredibly fast performance. Using external program ('cat') to index files in a directory (recursively), it indexes 180 files in 0.06 seconds. Query returning 44 results took 0.0015 seconds . Takes 24 MB virtual, 5.3 MB RSS-Shared. No GUI yet. I could have added a scheduler to pause for 10 seconds after every 10 files (5 more lines). This Lucene.Net based crawler and indexer beats beagle in performance but nowhere close to beagle.

Maybe beagle is not a lucene-powered locate. After all, to err is human.

And PLTC means ...

I just now saw the programmer's personality test webpage in Digg and decided to give it a curious shot. And ... my programmer personality type is PLTC.

(Update: The link to my personality doesn't work. I thought they have one static page for each possible combination. I am probably wrong, let me add the full text here)

You're a Planner.
You may be slow, but you'll usually find the best solution. If something's worth doing, it's worth doing right.

You like coding at a Low level.
You're from the old school of programming and believe that you should have an intimate relationship with the computer. You don't mind juggling registers around and spending hours getting a 5% performance increase in an algorithm.

You work best in a Team.
A good group is better than the sum of it's parts. The only thing better than a genius programmer is a cohesive group of genius programmers.

This one I like the best.
You are a Conservative programmer.
The less code you write, the less chance there is of it containing a bug. You write short and to the point code that gets the job done efficiently.

OMG. That's totally like me. Or whatever!