d-Tech-t

Aww... how he was loved

2011-02-13T06:27:00.000-08:00

I am guilty of orphaning the mighty Beagle. With so many bugs calling for attention, I reverted to debugging towards the end of the last decade; I had no option other than turning a blind eye to one of my favourite projects. There were technical reasons which acted as good secondary excuses - Evolution devised a new API for search engines which basically made beagle incompatible with that email client, the File system indexer (one of the many data gatherers supported by it) was full of corner-case hacks demanding cleanup and rewrite, and lastly, increasing attention to tracker, whose one of the taglines (possibly made up by fan-boys) was "C".

I havn't been catching up with beagle-tracker-mono world since then. To be honest, I have long stopped following /. and Linux news altogether. I couldn't get myself to unsubscribe from beagle mailing-list though, even though I was convinced within a year no one in the world would be using beagle any more and there will never be any email in that list. It was just matter of time.

Therefore, imagine my surprise to see the following recent thread in the mailing list. It was too nice and warming to read the posts once-twice-thrice. The stability and versatility that we aimed and delivered, is surprisingly, still relevant. Maybe I should title this post as "How he is still loved"

Thu, 10 Feb 2011 13:53:42 +0100
Dear beagle hackers,
since the mailing list is inactive I fear that beagle is now ultimately abandoned. Is this the case? If so, this really makes me sad. I have been using beagle daily for a number of years and I still find that there is no better desktop search available on Linux. Tracker is a disappointment. It supports very few data sources, is a cpu and memory hog, it is buggy, and its search results are very poor in quality, and I don't really see it making progress either. I got so used to beagle for quickly finding e-mail and files that I was saddened to see that Debian completely removed it from the archives, because development has stopped. Isn't there anyone anywhere willing to take over? There has been so much effort invested into beagle. In the end it was really stable, reliable and usable, that just letting it rot is such a wasteful thing to do...

Thu, 10 Feb 2011 11:46:20 -0600

I don't think anybody is actively working on beagle, but I wouldn't
call it dead.  It's still used by many people, and we're at least
keeping it runnable -- eg we had an update to adapt to new Mono.Sqlite
APIs upstream.  Feel free to work on it!

Fri, 11 Feb 2011 10:06:13 +0100

I would love to, but unfortunately I am in no way qualified to do so. It is good to hear that people are still using it, However, at least for Debian users this has become very difficult since it was removed from the repositories. Has it been updated to work with recent version of Evolution?

Fri, 11 Feb 2011 13:28:41 -0500

On Thu, 2011-02-10 at 13:53 +0100, Johannes Rohr wrote: 
> Dear beagle hackers,
> since the mailing list is inactive I fear that beagle is now ultimately 
> abandoned. Is this the case? If so, this really makes me sad.
> I have been using beagle daily for a number of years and I still find 
> that there is no better desktop search available on Linux.
> Tracker is a disappointment. It supports very few data sources, is a cpu 
> and memory hog, it is buggy, and its search results are very poor in 
> quality, and I don't really see it making progress either.

I feel the same way;  I'd hoped Tracker could serve as a replacement for
Beagle but to call it a disappointment is an understatement.  

Score a big one for the anti-Mono-FUD-monster  [funny they used to go
on-an-on that Beagle was slow because it used Mono... then Beagle got
fast and Tracker still devours gobs of CPU to produce essentially no
result].

But in Tracker's defense Evolution has been changing significantly in
the past few revisions so Beagle doesn't work with it anymore either.
In time hopefully Evolution will settle down again and the Tracker
integration will improve.  Some really nice improvements are being made
to Evolution.

> I got so used to beagle for quickly finding e-mail and files that I was 
> saddened to see that Debian completely removed it from the archives, 
> because development has stopped. Isn't there anyone anywhere willing to 
> take over? There has been so much effort invested into beagle. In the 
> end it was really stable, reliable and usable, that just letting it rot 
> is such a wasteful thing to do...

If there are no developers that is what happens.  Pointing out that
there "should be" developers doesn't make them magically appear.

Fri, 11 Feb 2011 13:47:15 -0500
>
> I feel the same way; I'd hoped Tracker could serve as a replacement for
> Beagle but to call it a disappointment is an understatement.

Ditto. I have continued to be disappointed by Tracker. I've completely
removed it from my systems as it was actively blocking other
applications. More trouble that it was worth.

Beagle was always very un-intrusive and always gave me great search
results.

Sad that it's moribund. :-(

Then there were posts about an alternate desktop search yada yada yada which I am skipping. Though, one thing should be kept in mind - one shouldn't apples and oranges. Beagle is not a file-system indexer. There are plenty of other tools to do that, and the simplest file system indexer can be written in less than 100 lines (of any modern programming language).

There is of course a file system indexer built on beagle that can be used to compare against existing file system indexers - it is called blocate and I believe, is shipped alongside beagle (in distros still packaging beagle). Let us bow once again to the stability and versatility - blocate is merely a wrapper shell script calling the regular beagle commands.

Superfish menu + Drupal Fusion + How the devil do I

2011-01-28T12:16:00.000-08:00

Superfish is a pretty cool menu based on JQuery and CSS. If you are suspicious, view the wonder for yourself at http://users.tpg.com.au/j_birch/plugins/superfish/ with the nav-bar example.

Fusion based Drupal themes are currently hot on market, e.g., Acquia Prosper. You can check them out for your own at http://fusiondrupalthemes.com/browse.

You are yet to hear the awesomest - jQuery hoverintent plugin. Very helpful for navigating complex menus, where often, one slip of the mouse implies restarting the navigation.

Fusion themes already use Superfish for their main menu system. Of course, I madly fell in love the moment I saw the navbar. However, the sophisticated CSS and JS of Superfish nav-bar goes kaput when used in Fusion. Quoting jeremy (a TopNotchThemer at http://fusiondrupalthemes.com forum)

Superfish is a bit difficult to work with when you need to get the active menu sub-menu to be visible when you are on that section. I've tried a LOT of different scripts and changes to the theme but never found a solution that worked for Superfish and it was very unstabble when I got close to what I wanted.
I actually went with disabling the primary menu and Superfish all together and going with a block instead for my menu. This is basically starting all over and doesn't work with Superfish at all so you have full control over the placement and styling but will have to write it yourself.
So for this specific case I recommend going with a block instead of the default primary menu with Superfish. Best of luck!

So, I started keeping notes when I saw some early signs of success. Here is the recipe to get it (close to) perfect.

Fix Superfish nav-bar menu in Fusion-based Drupal themes

1. Enable navbar for superfish
From http://fusiondrupalthemes.com/story/100525/customizing-superfish-menus-your-fusion-theme

function iiitd_preprocess_page(&$vars) {
  // Add Superfish navbar class if dropdown enabled
  if ($vars['primary_links'] && theme_get_setting('primary_menu_dropdown') == 1) {
      $vars['primary_links_tree'] = preg_replace('/^<ul class="menu sf-menu/i', '<ul class="menu sf-menu sf-navbar', $vars['primary_links_tree'], 1);
  }
}

2. Ask navbar to keep showing second level
http://fusiondrupalthemes.com/forum/using-fusion/adjusting-superfish-behaviour

--- mysubtheme.js

Drupal.behaviors.mysubthemeSuperfish = function (context) {

$("#primary-menu ul.sf-navbar").superfish({
    hoverClass:  'sfHover',
    pathLevels:  2,
    pathClass:   'active-trail',
    delay:       800,
    animation:   {opacity:'show'}, // opacity: fade-in, height: slide-in
    speed:       'normal',
    autoArrows:  true,
    dropShadows: false,
    disableHI:   false
  }).supposition();
};

3. Modify fusion-core/superfish.js DIRECTLY! Superfish removed the active-trail class which fusion didn't like at all!

return this.each(function() {
   var s = this.serial = sf.o.length;
   var o = $.extend({},sf.defaults,op);
   o.$path = $('li.'+o.pathClass,this).slice(0,o.pathLevels).each(function(){
+    $(this).addClass([o.hoverClass,c.bcClass].join(' '));
+     //.filter('li:has(ul)').removeClass(o.pathClass);
-    $(this).addClass([o.hoverClass,c.bcClass].join(' '))
-     .filter('li:has(ul)').removeClass(o.pathClass);
   });
   sf.o[s] = sf.op = o;

4. (As good as modifying superfish.js) Set the default value for Superfish directly - by the time drupal does it, the page is already rendered in an incorrect way.

--- mysubtheme.js

// Change defaults too - otherwise, second level does not show the active-trail
$.fn.superfish.defaults.pathClass = 'active-trail';
$.fn.superfish.defaults.pathLevels = 2;

5. Add space for the second level to show in the right place.

div#primary-menu ul.sf-navbar {
    height: 2.5em;
    padding-bottom: 2.5em !important;
    padding-left: 0 !important;
    position: relative;
    width: 100%;
    z-index: 97;
    line-height: 100%;
}

div#primary-menu ul.sf-navbar li {
  position: static;
}

div#primary-menu ul.sf-navbar li.sfHover ul {
  margin-top: 0;
  top: 2.5em;   /* match top ul list item height */
  left: 0;
}

6. Even though the theme comes with hoverintent plugin files, HI (as hoverintent is often called) doesn't actually work. Even after disableHI is set to false. The reason and the fix is ...

// Destroy default fusionSuperfish initialization.
/* Explanation: default fusionSuperfish disables HI during its initialization -
 * which, attaches a jquery hover() handler. The jquery hover handler attaches
 * mouseenter and mouseleave events to the element. The mouseenter handler first
 * actually changes the eventype to 'mouseenter' and then calls the handler.
 *
 * Now, HI requires handlers for mouseover and mouseout. However, upon firing
 * those events, first the hover handler is called. So, by the time the HI
 * handler is called, the event type has already been changed to mouseenter/mouseleave!
 * And that is how the cookie crumbled everytime.
 */
Drupal.behaviors.fusionSuperfish = function (context) { };
Drupal.behaviors.fusionSuperfishBlocks = function (context) { };

This post carries a timestamp of 1:30am at night. Don't even ask for warranty.

Internet to TV

2010-12-01T07:00:00.000-08:00

This is the age of parallel processing.

Multiple firefox profile running simultaneously (--no-remote / MOZ_NO_REMOTE=1).
Multiple tabs opened in yakuake.
Multiple CPUs (one for dealing with flash/firefox, other for life).
Now added, multiple sound outputs from computer.

I have one pair of speakers which I use in the dining area. Helps me get past barely edibles.

My work area has now moved close to the TV. This TV has HDMI (and is DLNA-compatible), so it made perfect sense to me when the geeky fairy whispered something about using the TV speakers for listening to Magic 106.7FM, Jango and Youtubish songs off of my not-so-great-sounding laptop. minidlna doesn't quite stream non-file contents, so I was left with pleading ALSA to let specially chosen audio to be routed for TV@dBera via HDMI.

$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: Intel [HDA Intel], device 0: AD198x Analog [AD198x Analog]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
card 0: Intel [HDA Intel], device 3: INTEL HDMI [INTEL HDMI]
  Subdevices: 0/1
  Subdevice #0: subdevice #0

$ cat > ~/.asoundrc.tv
pcm.!default {
    type plug
    slave {
        pcm {
            type hw
            card 0
            device 3
        }
    }
}
$ ALSA_CONFIG_PATH=~/.asoundrc.tv firefox -P music -no-remote

Hitting the right button on the TV remote sets its Energy Saving to Picture Off. Voila! I just got myself an awesome pair of speakers.

Edit: If you get no sound, make sure the IEC958 interface is unmuted (you can check/set/unset using alsamixer). IEC958 is the interface for digital output for a lot of soundcards. 

forever very-vapor

2010-09-03T11:37:00.000-07:00

WHAT ! Duke Nukem Forever is not a vapour ware ? But I thought that is part of the definition of vap...

I am feeling weak at my knees, being compelled to write about a hearsay read on blogs and twitter feeds. It's just that hard to keep cool.

Let's Nuke'em Duke!

this is his 4th computer

2009-06-07T06:01:00.000-07:00

A thin man with a pointed beard loved honeysuckle. He had some honeysuckle trees in an orchard across a huge river. He built a boat and used it to cross the river. The fruits were good. He ate some and sold the rest.
...
10 years later, the boat showed signs of age. He built a new boat. Life was good again.

1 millennium, 6 century and 5 decades later, a very smart man told us how to create a very smart machine. One machine which can do everything. Of course, the machine needed sophisticated instructions but there were a lot of people who can figure out the right instructions for any operation.

6 decades later. A company that sells computers made a wonderful laptop. A fat man who earns a lot of money bought one. The laptop performed top-notch, looked solid and meant business. He was happy.
...
2 years later, the computer slowed down. He bought a new one. The one felt as fast as the old one when he had bought it. He considered the purchase a terribly insightful investment.

200 years later. A group of bespectacled visionaries will figure out every detail of how our body works. A lot of companies will appear who will promise a live-long and live-happy life. "Don't like your middle ear? Here, we will replace it with a machine. 1-day money back guarantee". You can hear a lot in 1 day, so everyone will be satisfied. A young rich business couple will modify themselves to become what they want. They will get married. They will make it to the news as the first perfect couple.
...
7 days later, the man will get bored of his wife. He will find her pose unexciting and her actions banal. He will find a new woman for a huge sum of money.
...
After uranium, monitors and PCBs, the world will face the challenge of a new type of landfill: people.

Better than my best

2009-02-04T08:16:00.000-08:00

Wow! "Toshiba handheld hits 1GHz"
http://news.cnet.com/8301-13924_3-10155730-64.html

The news isn't that tomorrow's phones will beat my current best computer. Technology, like leaking water, always finds a way to go down ... err... advance. I like it or not.

What surprises me is the abusive intent of folks behind the technology. Like it or not, designing is more art than science. The kind of rationality and precision that goes into using the technology to create something more than a crapware is nothing short of the artistic choices a painter makes when rendering the perfect sunset over an island where he never went.

The minds that control the coding hands do not seem to realise this ... no matter what better hardware I come across, I also come across a newer version of the same software which makes it crawl. Even the upcoming dual-core 1.5GHz is not going to impress me much, not any more. Perfection is the key.

< Insert Your Favourite > Desktop Search Hackfest

2008-07-23T13:55:00.001-07:00

Good news - a (read: the first ever) Desktop Search Hackfest is being
planned after the Maemo Summit, in Berlin.
http://wiki.maemo.org/Desktop_Search_Hackfest

Not so good news - I will not be able to attend. It's not easy to
sneak out as a grad student from an US university (bonus if you are an
_alien_). Joe is not going either. Our in-house Xesam guru is about a
join a real job, the money-paying kind. It might be hard for him to
attend either. I dearly wish Beagle could somehow benefit from this
meeting. Good luck to the other projects. Get some work done and make
the users happy. It is nice to see that Strigi/Nepomuk devs are
attending. And thanks to Nokia for making this happen.

On a side note, look at the number of participants from the Tracker
project ! Wow, they're booming. Double kudos to them.

Creating new backend - help wanted

2008-07-16T20:47:00.000-07:00

Three years ago this month, the above was the subject of my first post to the
dashboard-hackers mailing list. I was writing a backend for Akregator. Its
immaterial what Akregator is; I was mostly pattern matching back then and
that is how I got familiar with C#.

I remember submitting a patch to beagle bugzilla a few days before that. It
was to filter JPEG JFIF comments because that is where digikam stored all the
descriptions. Now it uses IPTC tags. Does not really matter since beagle
indexes both.

Backtracking further, I got interested in beagle several months before that. I
had this incredible urge to add /author/, /title/, /subject/ tags to the PDF
research papers that I download and use beagle to search among them. The
first time I visited the beagle website and downloaded the tarball, the huge
list of dependencies really really scared me off. I used to allow more Gnome
on my desktop back then than I do now but nevertheless building beagle seemed
like a daunting task. I later freed beagle from many of its dependencies; I
feel that is my biggest contribution to this project.

I made a second attempt later; soon realized that even though I can get beagle
going, there was (is ?) no easy way to modify PDF document properties (in
linux). Then I switched my goal to index my pictures, which I have started
tagging and adding comments to, using my new found love digikam. Thankfully
the one-liner JFIF patch did not require much C# knowledge. It would not have
mattered anyway, C# is darn easy to pattern match.

After all these months, I still haven't managed to build my repository of well
tagged PDF papers (if I wait for several more months and stop wasting time
then the need would be gone forever). I do keep a static index for my
pictures but rarely search them. What a waste of time :-). Silly me!

As I am about to sign off this blog entry, I noticed that I was siging off
as "Bera" in my initial few emails. I wonder when did I switch to
my "signature sign" dBera.

Did I mention that beagle 0.3.8 was released some 48 hours ago. Go go get it.

One way to get things fixed in Ubuntu

2008-05-08T04:14:00.001-07:00

Become famous and then blog about it. Its easy (the blogging part). And it
works [1a, 1b].

Now if only someone famous blogged about some other literal showstoppers in
beagle and KNetworkManager, some more Kubuntu users would be happy [2].
Beagle is second class citizen in Ubuntopia, but I thought KNetworkManager is
important.

One thing I noted though, since Beagle was moved to Universe from Main, it is
getting better treatment. Periodic sync from debian is actually happening
unlike when it was in main; the core Ubuntu developers rarely found time to
update the Beagle package. Kudos to Masters of the Universe!

[1a]

https://bugs.launchpad.net/ubuntu/+source/galago-sharp/+bug/186049/comments/9
[1b] http://arstechnica.com/reviews/os/hardy-heron-review.ars/3
[2] https://bugs.launchpad.net/ubuntu/+source/beagle/+bug/207157

I knew searching would fail

2008-04-28T07:12:00.001-07:00

There is a great report about a usability test on the web about a guy
giving some common computer tasks to his girlfriend on a fresh Hardy
Heron installation. I found it via Slashdot but since fewer and fewer
people are slashdotting these days, so here is a link [1].

The tasks are well chosen and the user is _not_ a first time
grandma-type-user and approaches each task in the obvious possible
way. "Obvious possible" - for people not used to a Linux distribution,
where "doing stuff my way" rules over "getting stuff done".

As I started reading it, I knew at some point the user was going to
"search for something". And I knew she was going to fail. Which she
did.

The problem with most (all?) of the linux desktop search applications
is that they are cut out for a particular task and are (hopefully)
pretty good at it. Indexing is the keyword there - how to index all
kinds of data in the best possible way and then allow users to search
the indexed data. And there are lots of sophistication there.
Unfortunately the common search tasks by an user is not quite that.

- Search for a file by name - most common
- Search for files of certain types
- Search for files in home directory containing some text - slightly
advanced usage
- Search among browsed websites etc.
- As a computer "user", it is not clear why I would search for
websites in the desktop search tool and not in the browser. Of course,
once I am told this can be done in the desktop search tool _too_, I
would be extremely glad and nod in appreciation.

It takes time to write a desktop indexing and searching system. I
didnt believe it when I first heard of it and my friends asked me what
is so difficult about it other than implementing inotify. For some
reason it is. So a lot of effort in invested behind that. But there
has been less effort in presenting a failsafe, minimum capability
search experience in that direction. What do I mean by failsafe and
minimum capability ?

- One obvious way to launch the search tool (there could be more, but
there should be one which may not be the best but works in the worst
case)
- The obvious tool should never fail on the basic searching - never,
never, never. By basic searching I mean searching for non-file content
information - name, size, type (what on earth is _mimetype_ to a
non-CS/IT person ? searching by extension is what I mean. broad
classification like music, picture helps).
- Repeat the above. Let me say it this way - if the user knows a file
exists, she should be able to find it by name. And matches by name
should come _first_. Same for search by types.
- Anything else is a bonus. When we have complete semantic desktops,
where a file is same as an email and same as an addressbook entry,
maybe then users would want to search for everything or specify what
exactly she wants. Not now.

So where does beagle fall behind (or some of the others tools, by
reading about them and looking at their screenshots).
- User want to use them to search for files. The tools return pretty
much everything.
- Give them an option on where to search. There is no need to
include an option for "application/rdf+xul" but list the common
options. A search service has to work for the minimum, a GUI has to
cater to the average. I would be sad if it didnt have ways to cater to
the advanced crowd too but I dont mind if that requires one extra
step.
- User wants to search everywhere (in the filesystem).
- Thats definitely not what beagle was designed for. A beagle search
tool is not expected to do that. But when it is presented as the
_main_ search tool to the users, it will be used to search everywhere.
And it will fail.
- I dont quite know how to design a failsafe GUI search tool but a
good start would be use the indexing service to home directory files
and brute-force 'find /usr/bin' for non-home directory partitions.
- Some users would never need to search by content.
- If searching content was cheap then it would not be a big deal if
searching by content is enabled. But content searching is expensive
from my experience. It would be better if users are allowed to opt-in
for content searching.
- Content searching is not supposed to be expensive. As far as
beagle is concerned, it is halfway on meeting this goal. It still
needs some fault-tolerance feature to detect problems before too much
damage has been done.

There are lots of other ways to make searching "just work". The user
does not even need to know there is any indexing in the background.
The sad part is a lot of what I suggested (or could have suggested) is
already possible with the current beagle infrastructure. What is
lacking is someone with a good GUI knowledge to work on improving the
search experience. I am defending the base by fixing the occassional
simple bugs but a real developer is needed. And needed urgently
otherwise yet another distributions will be released with a lacklustre
search experience.

http://contentconsumer.wordpress.com/2008/04/27/is-ubuntu-useable-enough-for-my-girlfriend/

Takes Two to Release

2008-04-20T15:04:00.001-07:00

The GMail backend I blogged about before is now available for mass abuse in Beagle 0.3.6(.1). We also tried to maintain our love of cutting edge technology by upgrading the Firefox extension to work with Firefox 3.0.

I noticed several forum posts where users wanted to use beagle like locate/find-grep. The desire was two pronged - no intention to run a daemon continuously and return files from everywhere doing basic searches in name and path. That is not how beagle is supposed to be used but users are the boss in a community project. So I added blocate, a wrapper to beagle-static-query. Currently it only matches the -d DBpath parameter of locate but works like a charm. Sample uses
$ blocate sondesh
$ blocate -d manpages locate

The other thing I added was a locate backend. I absolutely do not recommend using this one. Yet if you insist ... when enabled and used with the FileSystem backend, it will return search results from the locate program. Yes, results from eVeRyWhErE, as you wished.

You can use both the GMail and the locate backends in beagle-search as well. But both the new backends are rather primitive, so I have taken enough precautions againsts n00bs accidentally using them. So in summary, 0.3.6 is not going to do you any good. Oops... did I just say that ?!

The title is based on the empiricial count of the number of actual releases (including brown bag ones) needed for last few releases.

You said Google owns your life

2008-04-08T12:31:00.001-07:00

And I nodded my head and felt sympathy for you. I also used the I_doubt_its_maintained_anymore and just_barely_works xemail-net to write a live GMail backend for beagle. It does not index the emails as of now, but uses the IMAP search protocol to directly search the emails on the GMail IMAP server. And searching followed by retrieving the headers of the matched emails is really slow; the delay is clearly perceptible. It could be due to xemail but I could not find a better alternative for a .Net IMAP library.

It should not be hard to take the current backend and add the ability to download a batch of emails and index them locally. Google also publishes a nice set of GData .NET API for accessing Google documents, calendars and a lot of other services. A backend for them would at least make our beloved maintainer happy.

A basic GMail query-only backend was on my TODO list for more than year. And now its finally done. Proper GMail indexing and Google service is also on my TODO list. So hope to see it sometime... say by next year.

Better late than never

2008-03-28T16:24:00.001-07:00

Beagle 0.3.4 was released into the wild last week. We tried to fix the build problems (new Gnome-sharp, Mono 1.9, missing files from last release) but building Beagle with Ndesk-DBus 0.5.2 remained broken, as per tradition.

Other than that this version builds nicely with Mono 1.9 and contains #ifdef-ed code to use the Mono.Unix.UnixSignal API when built with Mono 1.9. That should ensure that beagled and index-helper will quit when asked to quit. Yes, this is the 21st century.

There is a lot of mapping in Beagle and a lot of them are hardcoded. We are gradually moving them to user configurable files. The config files were moved out earlier; in 0.3.4 we have moved out the query mapping (mapping "ext:html" to right internal property name). If you want to add a mapping for some property you repeatedly query, just add it to the local ~/.beagle/query-mapping.xml

I added another handy tool to use Beagle as updatedb/locate. Use beagle-build-index or beagled to create indexes (like updatedb). And then use beagle-static-query to query the indexes (like locate) - no long running daemon beagled has to be running.

E.g. I created an index for system files in /usr/bin, /bin and other global directories (the --disable-filtering is to disable filtering file contents since here is only care about the file names and such),

$ beagle-build-index --recursive --disable-filtering --target ~/.systembeagle /usr/bin/ /usr/local/bin/ /bin/ /etc /usr/local/etc/

Then I could query just like locate,

$ beagle-static-query --add-static-backend ~/.systembeagle 'net*' --backend none

(--backend none is to tell beagle to not search any other backends). I could have added ~/.systembeagle to beagle using beagle-config so that I dont have to add this path everytime or I could have even created an alias for this.

Why do this when locate/updatedb already does it ? Because I can :). Ok, I actually use this to search monodocs. I am not a big fan of this mouse, point, click business and I stick to the terminal with mod and monop2 at my disposal. These tools are great once you know the fully qualified name of the method or the class. Use this jack-of-all-trade beagle.

Step-1: Enable the system wide monodoc index. Its one of the crawl-rules shipped with beagle but disabled by default.

Step-2: Let cron build it or you run the cron job yourself. Building the monodoc index takes time though. Definitely longer than any special indexer for monodoc files. But thats only a one time cost.

Step-3: Use beagle-static-query. You can also use phrases and wildcards '*' and search only in methods or classes or properties (just look in the returned fields and use beagle property query).

Beyond Search: arrhh...dee...efff

2008-03-08T13:36:00.000-08:00

If the news reports and blogs are to be believed, this is the age of Semantic Something. First people wanted to search web, then file contents, and then search emails and other user data. Everybody was talking about desktop search; along came Beagle, Spotlight, Google Desktop Search, Kat, MetaTracker, Pinot, Strigi etc. While desktop search at its core is nothing but a crawler which reads different file formats and stores them in a searchable database, searching is the most trivial and IMO, boring application built on Beagle's infrastructure.

These days the focus seems to have shifted to Semantic Desktop and Semantic Web. Most blog comments and mailing list posts about Semantic-Fu have a hint of it being vapourware. Its not totally their fault either; the ideas are around for a long time and people are working on it for many many years. But there is no glittering gold in sight. Only recently some interesting Semantic Web ideas have started taking shape. Semantic Desktop is a slightly different game but it should not be far behind. After taking about 40 developer years, Beagle is just about ready to take desktop search beyond simple file content search. Historians might want to take note of the dashboard project and how beagle came into being as a necessary requirement for that truly beyond-desktop-search application.

The core idea behind Semantic Desktop, upto my understanding, revolves around the buzzword jack-of-most-trades RDF. And for the impatient kind, here is a rude shock - RDF is not useful for human beings. Even further, it is not even meant for you, me and us; storing every conceivable data in the RDF format is not going to make our life any easier right away.

RDF or Resource Description Framework is a generic way to describe anything, to be accurate any description of anything. It is a fairly elaborate yet structured format; very easy for programs to analyze that information but extremely redundant to human eyes. Notwithstanding what the AI experts are claiming about the future of AI, human mind can work without immediate deductive reasoning and in fact does that a lot of time. It recognizes familiar words without reading the alphabets one at a time, it deduces the color by merely glancing at it, it conjures up strange connections; its a wonder that will be hard to completely characterize by any set of rules. At least at the current stage, algorithms have to be told the facts and the relations between them for them to do any kind of processing with its data. These are the things that we just know when we see something and is thus the reason why storing the description of something in an RDF format is not going to gain me anything immediately. On the other hand, this is also the reason why applications should be fed data in an RDF format to allow it unhindered access to the semantics of the data.

If that felt hand wavy, try to think about the difference between the semantics of a data and its syntax. An array could be used to represent a linked list, a queue, a stack, a tree or an heap - the latter are the different semantics of the representations, the array is one of the many syntactic representations of one of the latter concepts. A bunch of pairs could be stored in a database table; the table is a syntactic representation of the data which has the semantics of a bunch of name, phone-number pairs. It is hard to work with the semantics of an idea, in a sense it is something up in the air; on the other hand storing some data in a suitable working form could fail to capture some concept about the data. Also, once stored in a particular form it is easy to miss the bigger picture; thus limiting the scope of what we could do with that data.

Saying all that, for the time being think of the RDF format as a bunch of objects and facts where each object is related to some number of facts. The semantics of related could differ based on the context, and RDF is powerful enough to describe even that semantics and a whole bunch of other facts about the facts. With beagle pulling data from nooks and corners of a user's desktop and providing a service which allows applications to search this data, it is a shame if we cannot exploit the relationships in this data for a better mankind... err... dolphins... err... us.

Consider all the emails I have. Now I know that there some emails that are part of discussion threads. Beagle does not. With the beauty of N3 (a close cousin of Semantic-Fu and RDF), I can write this extra information as a set of rule (the single '.' represents end of one rule). I am using emails msgid to track emails in a thread. I could not help but notice the similarity of these rules with prolog or other logic programming languages.
/* an email with subject 'foobar' is in its own thread */ { ?email :title 'foobar' . ?email :msgid ?msg . } => { ?msg :inthread ?msg } . /* if any email refers to some email in thread, then this email is also in other email's thread */ { ?ref :inthread ?parent . ?email1 :reference ?ref . ?email1 :msgid ?msg .} => {?msg :inthread ?parent} .
Using the RDFAdapter of the beagle-rdf branch, I can use this to get all the emails in the thread with foobar in its subject. Note that I am able to write my set of rules only when I see this data as actual emails and not a bunch of lucene documents with fields. The latter carry no meaning. Further note that, I can also use the BeagleClient API to perform field specific queries to obtain the same results. The difference is that the process of using BeagleClient will require me to think about the relationships from scratch and then figure out the right sequence of queries. Instead I could store all the relationship among the emails in the email-index in an RDF format (and also related information not stored in the index e.g. saying a list of email addresses are all mine and should be treated as for one person). Then, whenever I want to extract some information, I can write the question (again in an RDF format) and let the RDF-Magic figure out the how to execute this question against that data given this set of inference rules. Isn't it cool ?

If I missed it earlier, this kind of data-mining operations are not for my everyday use (here my refers to usual computer users) and is not for everybody. Still it is can sometimes come in handy. Imagine the possibilities if you can write the relationships between a file in an mp3 playlist (playlist filter), its download link and how your arrived at that page (webhistory indexing), the email you sent with that file as an attachment in a zip file (email attachment and archive filter), its ratings and usage statistics in Amarok (amarok querydriver) and of course the actual file on the harddisk (user home directory indexing).

Warning: The RDF Adapter in beagle uses the sophisticated SemWeb library which allows anyone to perform graph operations (selecting subgraphs, walking on graphs, pruning nodes and edges etc.) on the RDF graph of the data. Unlike most RDF stores for desktop data, beagle is not optimized for RDF operations and could take quite a bit of time and heat up the CPU. It took me about 4 seconds to find all threads with the word beagle among 500 emails (my actual email index has about 20K emails! I refuse to imagine what will happen if I run it on the full index). If you are interested, checkout the rdf branch and take a look at the test SemWebClient.cs.

beagle searchhandler for khelpcenter

2008-02-12T17:25:00.000-08:00

I realized that it is extremely easy to build searchhandlers for KHelpcenter. I have always wanted to use beagle for searching in KHelpcenter, so I quickly made a searchhandler in python. Screenshot below. To use it, put these files in the right places, chmod khc_beagle.py to make it executable and restart KHelpcenter. This only searches the manpage index for now and can be easily extended to search docbook. And of course you have to remove /usr/share/apps/khelpcenter/searchhandlers/man.desktop. I would submit it to KDE bugzilla but I dont know if this works with KDE4 version of KHelpcenter.

Grab the 3rd of the third

2008-02-02T19:16:00.001-08:00

I announced the release of Beagle 0.3.3 today. It is one of those releases with lots of features here and there which just makes me nervous. I found a problem (which incidentally was uncovered when we started using Sqlite prepared statements couple of weeks ago) today during. I quickly identified the problem and fixed it but I am hoping everything else works out OK with this one.

Apart from using Sqlite prepared statements, which should show some speed improvements during running beagle-build-index, there are few other goodies as well. Beagle-search includes a menu option to view the current index information. I would have liked it better if it kept refreshing but this is better than nothing.

Searching documentation is now enabled in beagle-search. It used to be disabled by default in the early days because apparently it returned a lot of results and messed be Best. The situation is better now but not by much; so you have to pass --search-docs to beagle-search to ask it to search the documentation index. That aside, system wide manpage index is now enabled by default and that includes lzma compressed manpages (so Mandriva users like yours truly will be extremely delighted). Beagle search happily searches manpages and it is a real pleasure to use it instead of man -k.

Another real pleasure is being able to create Qt GUI using C#. Its feels very good and I ended up creating beagle-setting-qt, a Qt GUI for beagle-settings. I never added the bindings for the new beagle-config to libbeagle. So I had to amend my fault by giving KDE users some GUI for beagle-settings.

You also get a fake implementation of searching in a directory, one of the popular requests. You can either search inside a directory by giving its full path or by giving a word in the directories name. One catch is that the search is not recursively under the directory but only in its contents.

Sadly almost all the currently known problems with beagle are outside our control. Fortunately, most of the problems in these dependent libraries or suites are fixed and will be released soon. It is surprising how bugs in these external programs, generally corner cases when used alone, are triggered when using beagle. A long running application for desktop users has to cover a lot of bad ground to be even slightly respectable.

If this release goes well, then we might try to fix all the horribly hacky property names (based on our own ontology) and come out with a 0.4.0. I am also hoping to merge the RDF branch to trunk before that. I should really blog about the RDF branch sometime; the experiment to overlay an RDF store on beagle's data is nearing a sure success.

FOSS meets ENG

2008-01-29T10:24:00.000-08:00

Techkriti brings back fond memories. Techkriti is the annual technology fest at IIT-Kanpur (y'no the place where they figured out how to decide if a number is prime in a theoretically fast way).

That was one fabulous event I actively took part in during my undergrad days. A perfect mix of all kinds of technology. I was there in its early days of infancy; if I remember correctly the first prize for the software contest in my first year there went to a graphical calculator program in Tcl/Tk. By the time I left it was hugely popular and there participants from all over India. I was one of the organizers for Tech Olympiad in my final year. It was equally fun to come up with challenging problems where various concepts tie in together. The participants loved it.

This year Techkriti is even more exciting. I read in the news and blogs about how FOSS is catching up in the subcontinent. This time they are organizing a FOSS event for Techkriti, probably for the first time. I am hoping the event becomes a success, though the schedule page is a bit empty now. Beagle Xesam adapter author Arun and Web interface dude Nirbheek are among its organizers, so I am sure I will get first hand information about all the exciting things that will happen (how about a hackfest for Dashboard ;-).
.

I will end with two simple brain-stormers.

The first one is of mathematical type, my favourite: Show that any 5 consequtive numbers will always contain some number which is prime to the other 4 numbers.

The second one relates to programming (somewhat), my hobby. You are given two arrays A1 of size m+n and A2 of size n where only the first m slots of A1 are filled with sorted integers (increasing order) and all the slots of A2 are filed with sorted integers (increasing order). You goal is to merge the two arrays into A1, in linear time (no block array copying or other tricks) but (here is the twist) using no extra space (i.e. no placeholder variable to hold temporary values).

Good luck!

Open letter to OpenSUSE users

2008-01-25T17:39:00.001-08:00

(long post warning)

Dear OpenSUSE users,

Recently I came across several threads in various OpenSUSE mailing lists
[1], [2], [3]. I was both amused and felt sorry while reading the posts. No
really some of you write funny emails. That aside, people, especially those
using FOSS don't make up things like this. I am sure the problems that you
faced exist (or existed in whatever version you were using).

I joined the project later but I still feel responsible for the sleepless
nights some of you have had due to beagle, trying to imagine what you would
see beagle has done to your computer when you wake up. I would have felt the
same if I were in your position, in fact I sometimes feel the same for one of
the browsers that I use.

There were lots of suggestions and speculations. There were suggestions of
filing bugs with us. While I do appreciate if some you can file bug reports,
I sympathise with those who dont want to open yet another account to file
bugs or email the mailing list. I belong to the latter group, so instead of
replying to the thread, let me take a minute here explaining how we try to be
friendly to your computer hard-disk space, memory and CPU.

* We nice the process and (try to) lower the iopriority.

* Extracting text from the binary files, without rewriting the app which
deals with files of that type, is an expensive operation. So, we index them a
few at a time with sufficiently long wait in between. The wait period is
longer if the system load average is high. But if you are playing games or
doing other CPU intensive operation, you will not miss the CPU spikes. Normal
uses should not be hampered though.

* During crawling (for files, emails or browser cache) we try not to disturb
the existing vm buffer cache.

* We believe once the initial indexing is over there should not be noticable
effect from beagle, so we crawl a bit faster when the screensaver is on. But
we provide options for you to turn it off.

* We use a separate process to actually do the dirty job of reading the
files and extracting data. As a failsafe measure, if the memory usage of that
helper process increase too much we kill it and start a new helper process. I
would like to claim that for the last several versions I did not see/hear the
helper process being killed due to memory problems.

* For certain files we need to store indexable data in a temporary file. We
make sure we delete it as soon as the indexing is over. There were problems
in some old versions where the files would not be deleted (they definitely
wont be deleted if you kill-9 the process) but I have not heard about this
problem in recent times.

* To show you snippets of your searched words, we store the text data of
some types of files (not the text or c,c++ kind of files whose text data is
basically the file itself but the files in a binary format). We try to be
smart here to not create thousands of small files on the disk (I have about
20K mails generating at least 10K snippet files). In addition to it, we
provide ways for you to turn off the snippets completely.

We do care for your experience and certainly for my own experience while
indexing my data. So where do we go wrong:

* Once in a while the indexer encounters a file for which it ends up in an
infinite loop. Most of the times it is generally a malformed file but
sometimes it is also our fault.

* C# has lots of advantages and one of them is that the developer does not
have to worry about freeing memory after it is used. Depending on someone
else (in this case the garbage collector which frees the memory for us) has
its pros and cons. But one thing for sure (assuming mono is not making any
mistake in freeing), there is not going to be any memory leak of the kind we
are afraid of in C or C++. Neither are we afraid of segmentation faults due
to memory corruption. If you are wondering how do some of you see beagle's
memory growing, let me remind you that "to err is human". With sophisticated
tools to prevent simple errors, comes sophisticated errors. A simple example
could be like storing in a list all files beagle finds during crawling, but
forgetting to remove them once the data is written to the index. No, we never
did that but sometimes we make similar mistakes.

* We would be extremely happy if beagle only used C# for all its operations.
Unfortunately, we have to depend on a lot of C libraries for indexing certain
files. Sometimes memory leaks (the C type) and segmentation faults happen in
them. These are harder to spot since mono does not know about the memory
allocated in the C libraries.

* Beagle re-indexes a file as soon as possible once it is saved. It is in
general not possible to know whether it was the user pressing ctrl-s in
KOffice or a torrent app saving the file after downloading one chunk of data.
As a result, beagle performs horribly, yes horribly, if it encounters a file
that is being downloaded by a p2p/torrent app. You are bound to see almost
continuous indexing as beagle strives to index the updated file for you in
real time. Same goes for any large, active mbox file in the home directory
_not_ used by Thunderbird, Evolution or KMail (for mbox files of these apps,
the corresponding backend is smart to index only the changed data).

* NFS shares have their own share of problems with file locking, speed of
data access etc. We have tried to deal with them in the past by copying the
beagle data directory to a local partition, performing indexing and then
copying back the data directory to the users home directory. It is a feature
not continuously tested and I am sure you can think about lots of cases where
this would fail.

* The first attempt to write a Thunderbird backend was a disaster. Well, it
was good learning experience for us but it will cause headache to most users.
We disabled it in the later 0.2 versions. There is a new one in the 0.3
series which reportedly works better.

* There was one design decision which backfired on us. Imagine you dont have
inotify and have a large home directory. To present you changes in real time,
one option is to crawl the directories regularly (kind of what the WinXP
indexer does). You can imagine the rest. Though inotify is present in the
kernel these days, the number of default inotify watches (the number of
directories that can be watched) is pretty low for users with a non-trivial
sized home directory. In the recent versions, we disable the regular
recrawling.

* Besides continuous CPU usage and hard disk activity (for days and weeks
after the initial indexing is over) the above had another effect on the log
file. Add to it the pretty verbose exceptions beagle logs. We want to know
about the errors so we still print verbose exceptions but we dont reprint the
same errors anymore. (I have been told that some of the OpenSUSE packages
have the loglevel reduced to error-only which will automatically generate
smaller log files).

* This is a good excuse to end the list with. C# and beagle architecture
allows us to add lots of goodies. After all, we (read I) work on beagle
solely because I love to play with it. The more the features, more the lines
of code and more errors. The only good part is once spotted, they are easy to
fix. Check our mailing list and wiki for the available freebies.

So in summary, we try to be nice to your computer (and to you ? maybe, if you
are nice ;-) ... just kidding) but there are limitations that we are
constantly trying to improve on. Any of you can look in our bugzilla, our
mailing list archive, our wiki or hang out in our IRC channel to see for
yourself how we try to issue any problem with utmost importance. Ok, I lied,
as much as our time permits. There are lots of features in beagle and some of
them rarely get regular testing, mostly because none of us use those
features. I wont be surprised if there are major problem with these. I assure
you that if you bring any problem to our notice, they will be taken care of,
if not completely resolved.

Lastly, I read in one of the forum posts that beagle-0.3 will land in factory
sometime soon. If any of you wants to verify the facts above (the good ones
or the bad ones ;-), give that a spin. And a friendly suggestion, if you only
want to search for files with certain name or extension, you can do much much
better with find/locate.

Your friendly beagle developer,
- dBera

[1] http://lists4.suse.de/opensuse-factory/2008-01/msg00157.html
[2] http://lists.opensuse.org/opensuse/2007-12/msg01796.html
[3] http://lists.opensuse.org/opensuse/2008-01/msg01083.html (could not find
the parent of this thread)

klik2 klik beagle

2007-12-28T18:59:00.000-08:00

Yay! I managed to make a klik2 recipe for beagle. In principle this should enable anyone to just do
$ klik klik2://beagle
and happily run beagle. All the dependency packages will be automatically downloaded and managed in the background. Or to download once and use many times, you can do
$ klik get beagle
$ alias runbeagle='klik run ~/Desktop/beagle.cmg'
$ runbeagle

Not all of the above is happening right away; klik2 is under development and is looking promising but not completely done yet. But if you already have mono and want to run beagle, you can sort of do it now. This works for any distribution that klik2 works for, which is basically almost all the major distributions.

Get and install klik2
Download http://cs-people.bu.edu/dbera/blogdata/beagle.xml
$ klik get beagle.xml

It will do a lot of stuff and end up failing since there is no single app called "beagle" in the beagle package. It will however create the file ~/Desktop/beagle_0.3.1-2.cmg

Run beagled.

klik run beagled ~/Desktop/beagle_0.3.1-2.cmg --fg --backend manpages --backend Files

Browse to http://localhost:4000/ to access beagle using the web interface. You can use it to search, check indexing status and shutdown beagled.
To run the other beagle tools, the pattern is the same

$ klik run beagle-command ~/Desktop/beagle_0.3.1-2.cmg beagle-command-params
For example, to shutdown from command line

$ klik run beagle-shutdown ~/Desktop/beagle_0.3.1-2.cmg

The next time you want to run beagled, you need not run the recipe again; start from step-4 straight away!

one zero-three-two sailed today

2007-12-27T14:11:00.000-08:00

Beagle 0.3.2 was released today. On one hand we are still catching up with the regressions and new bugs that were introduced in the mighty beagle-0.3.0 and on the other hand, new features are streaming in. While yelp remains broken with beagle, I was amazed at how easily I can search within the manpages and double click on the results to open them in yelp. A much better alternative to man -K.

In other news, Lukas has started working on providing spelling suggestions Did you mean ... ? There are some technical limitations which are not fully resolved yet so it did not make it into 0.3.2 . It is currently housed in a branch and I hope to release it into the wild soon.

Beagle was not designed as an RDF store at its inception. It will take quite some work to make it a genuine RDF store. But what if there was an RDF adapter that sat between an RDF client and beagle and talked to each of them in their corresponding language, yet maintaining sanity. There is an ongoing work to overlay a Semweb selectable source on top on beagle. We will see how that goes.

A post on klik2 rekindled my desire to create a klik package for beagle. It will be easier this time since klik2 handles command line programs. I tried the automatic debian generated recipe and it mostly worked. Mostly, because one of the tools in beagle ran with --version and --help but failed to find some libraries to do anything more. I think all I need to do is to teach klik how to set certain PATHs and environment variables. Pretty exciting, what do you think ?

Enterprise search OR How to index on-demand

2007-12-13T18:44:00.001-08:00

If you are like me who keeps their filesystem organized, have a relatively
unchanging home directory or just simply do not want realtime indexing, you
can use beagle-build-index to meet your needs.

Beagle-build-index builds static indexes from files. Static indexes are
created and updated on demand when beagle-build-index is run but the
directories are otherwise not monitored for changes. The next time the
command is run, the changes are registered in the index. Once the static
index is created, you can ask beagled to search in it (by
passing --add-static-backend /location/of/static/index). beagled need not be
stopped while running beagle-build-index, it will automatically use the
updated index for searching once beagle-build-index finishes.

That was for files. If you want to do the same for anything else, say emails
or notes or addressbook and you do not want realtime monitoring, start
beagled normally and let the indexing finish. Then stop beagled and restart
with the parameter --disable-scheduler. Unfortunately, to update the index
with changes, beagled needs to be stopped, started normally and allowed to
run till updating of index is done, stopped and then again started with that
parameter.

If you are a system administrator managing lots of users and you dont want to
run beagled in realtime indexing mode for all of them, you can use the above
procedure to create/update static indexes, say once a day.

Or if you don't like mono, you can use Recoll for files and mairix for emails.
There are probably many more such tools but these are the two I know. Just in
case you have not heard about Mono, it is an open source implementation of
ECMA standard compliant C# compiler and a Common Language Runtime. And some
more goodies, all in all pretty useful.

Many reasons to like, what's yours ?

2007-12-12T19:01:00.000-08:00

Beagle 0.3.0 was released beginning of December. It is nearly 2 years since 0.2.0, more than 10 months since the last feature release and it has been about 2 weeks since then. In the mean time we identified some problems upgrade problems with 0.3.0 and released 0.3.1 and Mono released 1.2.6.

In contrast to 0.1.0 and 0.2.0, beagle-0.3.0 did not have any single major-impact change. But there were lots of small changes, all over the summer months and the months following them. It was getting increasingly difficult to handle all the small changes without going through the "Release early" trick and at some point we paused development, did a test release and then finally released what we have as a major release. I am personally expecting a fair share of bugs and regressions.

What are these small changes anyway ? I will leave out the invisible ones, some of which I have blogged about before, and only explain the ones that will directly impact your desktop usage.

There are 3 new backends: the Thunderbird backend (newly written, much better than the earlier one), the Opera history backend and the Nautilus metadata backend. There is also the TeX filter, one of our most demanded ones and new audio filter based on Taglib-sharp. There are new Firefox and Epiphany extensions which do a lot more than indexing browsing history and bookmarks.

The UI got some love; specially a bunch of useful options were added to beagle-settings like the backend selecion list. For obvious reasons, users should disable the backends they are never going to use.

One of the side effects of the beagle textcache previously was the creation of thousands of small cache files on the disk. People reported that the external fragmentation was wasting a lot of space. The textcache module was redesigned to minimize the fragmentation; I am sure you will appreciate the recovered space. We also compacted the external attributes; besides other benefits that will save some more space.

Two major enhancements were made to the query syntax, which is already quite rich. Date queries are now possible; date queries do not make complete sense without date range query, so that too is possible. And a new "filetype:" keyword was added e.g. to search for images use "filetype:image", to search among documents use "filetype:document" etc.

The major complains against beagle are constantly high CPU load, high memory usage and improper termination (or not exiting at all). The first two are well known and oft discussed. The third problem is not directly brought up, but have been found to be the reason upon close investigation. I gained valuable experience trying to find my way through the web of signals, threads and events in beagle code; a number of key issues were spotted and fixed. Oh, and the first two issues were also dealth with, as much as we could diagnose, but that is nothing new. It will sound funny, but a few of the high CPU and memory problems are direct results of some of our decisions that backfired. Some of them were fixed and the others being worked on.

2 experimental features were also added. One is a web interface to search beagle from Firefox (gecko based browsers really). You can also create standard bookmarks for common search terms. The neat thing about this web interface unlike the earlier webservices based one is that there is no heavy weight server running on beagle's side. This one communicates with beagled using BeagleClient XML based API and builds the entire GUI on the client side; a pure Web 2.0 AJAX/XSLT/CSS webapp (ok, these are some cheap buzzwords).

The other fancy feature is searching other beagle daemons over the network. Using Avahi you can even publish your beagle daemon or discover other beagle daemons in the network. We haven't quite figured out how to handle security, authenication and some other issues. So the feature is disabled by default and marked as experimental but I believe it can be used in some innovative way.

We received request from some distributions about global config files; useful for both distributions and sysadmins. Some useful global configuration settings would be to exclude certain directory from indexing for all users, adding or removing file ignore patterns from the default list, disabling of KDE backends by default in pure Gnome distributions. Some of the options were moved from the code to the config files, so that they can be set globally and overriden by individual users.

These are only some of the major ones.

Lastly, the reason I got excited about mono-1.2.6 is because it has some fixes and improvements that will be directly visible when using beagle.

Hide and Seek where I lost!

2007-11-08T14:36:00.001-08:00

Once upon a time someone created a Bittorrent website named SeedPeer [1]. Over
the years they grew bigger and popular. Then they also wanted to write a
bittorrent client. Recently they announced that their client (named SeedPeer
too) is mostly ready.

The above part is known. Then, Alan, the author of very-much-open (MIT/X11
licensed) MonoTorrent posted his objection that SeedPeer was based on
MonoTorrent (is it a rip off ? or based on ?) [2]. What annoyed him was that
the SeedPeer folks said nothing about MonoTorrent anywhere and claimed that
the code is all theirs. Quite right for Alan to be disappointed. I read his
blog, chuckled and said to myself ..."sigh! yet another victim of MIT/X11 and
BSD style licenses"[3] (coincidentally, I have read several posts in the last
few weeks that somehow end up with the openness of the above licenses; BTW,
beagle is also under MIT, which means you can sell it too...).

I like playing hide and seek so I downloaded SeedPeer to see how they managed
to agree to MIT/X11 and not say anything publicly. Quite possibly they credit
the author in a sneaky way somewhere. Ahh no. The exe and other related files
mentioned nothing of that sort. Ok, I went ahead and posted a comment
pointing out that SeedPeer people need to abide by the license [5].

For some reason I had that page open, so when I checked later a lot of people
were complaining of plagiarism and stuff. Probably someone even claimed that
SeedPeer has nothing do with MonoTorrent (it was something like "just because
its in C# does not mean ..."). I said to myself valid point, maybe Alan is
wrong... and thought to perform a silly test: compare the output of "strings"
of SeedPeer.exe [6] and MonoTorrent.{exe,dll}. Hmm... there was a lot of
matches, not only coincidental ones but suspicious ones. Like SeedPeer.exe
had "Mono.Math" in it - why was SeedPeer using Mono if they claimed before
that they are only focussing on Windows for the time being. Then there were
lots of strings that I only found in MT source and with uncommon names. I
posted a message in jest, saying I found a huge list of common strings and
that they probably have a few minutes to mangle the class/method names in
their source.

It was quickly followed by another post (from one of the admins, I suppose)
that they are not afraid and I should post the list. Also once their software
is complete, they would open source it. That kind of ticked me off ... could
I be that wrong ? Could the common words in "strings" be that coincidental ?
Well, off to "monop2 -p", the easiest "decompiler" at hand. Got the names,
removed the namespace and did a direct comparison. Irrelevant, but I could
not find any one line shell command to return the common lines from two
files; google helped me find a one line perl command. Anyway, there were 109
matches! And a lof of them were from private classes in MonoTorrent. The
similarity was too striking to be a coincidence. I modestly reported my
findings.

I have not been able to access the site since then. It was down; I should say,
it was down from my computer. Because, curious, I accessed the site from
another computer and it was nice and jolly up. Except ... the SeedPeer.exe
article, the download link to the exe and some other news articles are gone
[5]. I guess they "banned" me for, I guess reading their policy on their faq
page, "using comments section to insult others" (that was the closest match;
I also could not figure out if they mean blocking IP when they say "banned" -
since I am not a member at all).

I wished I had accessed that page using Firefox. Beagle firefox extension
would have indexed it right away and I would have a copy of the forums page.
I used Konqueror which does not always save the pages in its cache directory;
I just checked, there is no cache file for the forums page.

Like Adleman [7] (ahem), I like to break stuff. And thats what I did when they
argued that SeedPeer is their own and solely theirs. But I suck at
hide-and-seek, I really do!

[1] http://www.seedpeer.com
[2]

http://monotorrent.blogspot.com/2007/11/i-heard-some-great-news-earlier-in-week.html
[3] http://www.opensource.org/licenses/mit-license.html
[4] http://beagle-project.org
[5] http://seedpeer.com/sitenews/article/31.html
[6] http://cs-people.bu.edu/dbera/blogdata/SeedPeer.zip
[7] http://www-groups.dcs.st-and.ac.uk/~history/Biographies/Adleman.html

Good Luck Joe

2007-11-02T14:46:00.001-07:00

Please join me in welcoming Joe Shaw, the second non-Novell maintainer of the Beagle project. Joe has solid knowledge of the codebase yada yada yada...

KGiMAP

2007-10-24T11:32:00.001-07:00

A not so beautiful Allston morning turned out to be a rather useful one with the slashdot post [ Free IMAP On Gmail]. I checked my gmail settings and (apprently I am among the fortunate ones here) I saw the "Enable IMAP" setting there. Wee...

Fired up Kmail, added a new account for Google as a disconnected IMAP. And on my way to sync ... I was curious how the labels were going to be handled, I have about 15 of them based on which mailing list or which bugzilla an email is from (I have mentioned this before - I prefer organization over search). It was nice to see the labels getting downloaded as imap folders. And there is a special folder [Gmail] which contains the regular gmail folders - All mail, Starred, Trash, Spam, Sent Mail. Thats when it struck me!

All the emails are going to be duplicated or triplicated or could have even more copies. Labels are not disjoint, so there are mails in my Inbox that are also in beagle-bugs and of course all mail ever is also in All Mail. To add to the trouble, deleting an email locally is going to archive the email (basically, it removes the Inbox label from the email) and not delete it. The huge amount of mails in the Spam folder is not healthy for my hard disk space. The huge amount of emails my beagle-bugzilla filter sends out to Trash is similarly bad.

As of now I resolved the situation by asking all my filters (I have one filter for each label) to archive the email ( Skip the inbox) and then apply the label. I asked KMail to not subscribe to any of the [Gmail] folders. And I created a Gmail label(IMAP folder on KMail) Totrash which I marked as my Trash folder in KMail (for the GMail account). Now whenever I delete a Gmail email from KMail, it will go to Totrash which I can empty periodically from the web-interface. What a kludge! I wish there was a rule to empty folders or a filter move emails from Totrash to Trash!

There is no third way

2007-10-13T12:00:00.001-07:00

(This post is not about beagle though it is hosted on PlanetBeagle.
However it is somewhat related to "search".)

I have seen two approaches to finding information, in land of software
as well as real-ware. There are people who find it themselves and
there are some who use an external agent to get what they need. Human
memory is expensive, so these are the most obvious approaches. I have
seen equal number of people on both sides.

In the first group are people who like to keep everything organized,
from physical files to computer files, books to RSS feeds, knives to
applications. Organization typically involves grouping the items into
categories and re-grouping again if the number of items in a category
grows too much. Its a pain to remember things, then there is so much
to remember. Care is taken so that the number of categories don't grow
too large or the nesting isn't too deep. Computer Science people
cashed in on such problems, analyzed them to death and presented us
with databases, shallow and deep trees, hashtables, sorting etc.
Whatever the approach, the objective remains that reaching one
particular item should take as less amount of memory and labour as
possible. Memory and labour appear as trade-off here, so there are
variations of this where the most frequently accessed items are
optimized more than others (compression, source coding, amortized
algorithms, cabinets with upper and lower shelves etc.).

The second group of people use an external agent to do this task. In
reality, not everyone belongs to only one group, people tend use
whichever approach is suitable. Anyway, the external agents could be
one's mom, secretary :), a database, a filesystem, search engine etc.
They are nothing but efficient members of the first category. The
efficiency matters here; tolerance gets extremely low if I am not
doing something but hiring someone to do it.

It is clear that when dealing with items on the start menu, the same
ideas would come into action. OK, so the rest of the post is about
start menu, you can stop reading now.

Either we have a menu, properly organized or we have a dump of
everything with an extremely efficient way to get what we want. OK, we
can do the optimized variation of the first and add search to it to
make it work for the second group (e.g. kickoff, internet with search
engine and directories, filesystem with desktop search). But to get it
done right, both should be done with precision and care. Which means,
the search has to be fast and accurate and categorization should not
be too deep nor too flat. Geeshh... that is difficult.

Actually not that difficult for applications. If done the right way,
they can be arranged in a start menu with categories (not Windows
style though!). Or they can just be there and you find them using a
search tool (quicksilver, katapult, deskbar). Or both (kickoff,
kickoff spinoffs).

This is where I found Mandriva to go wrong. They had a nice start menu
to begin with (because Mandrake had a nice start menu to begin with).
In the latest distribution, the Mandriva specific categorization has
been replaced with a two level (i.e. top-level with
network/tools/office and then apps in them) menu. Some of the submenus
became too large, and they have been put behind another nesting called
"More" (e.g. "Firefox" went behind more because it not native KDE).
The end result is a menu, which is only two level and pretty flat (so
less to remember which submenu to go to) but once in the submenu, have
to memorize the position in a pretty long vertical list. Add to that
the effort in dragging the mouse a significant vertical distance. I
have entries that requires me to drag the mouse all the way from top
(I have my start menu button at the top-right corner) to the bottom -
that's 1024 pixels. The earlier menu was the same with one more
nesting, like "Browsers", "Spreadsheet", "Terminals" where I only had
to memorize the levels and then look in the 2-3 items in each
immediate category. The vertical distance covered in the current menu
is larger than the horizontal distance due to that extra submenu.
Adding to the woes is the lack of an embedded search option.

There is another aspect with menu which make vertical movement much
harder than horizontal movement. Its related to Fitts's Law. But first
a small digression. I like to call it Fitts's Theory. Because it
describes certain operation in a particular model. Calling it law
isn't wrong, except it makes it sound like anyone not following the
"law" is from Mars. Once again, it tries to model some observed
phenomenon, and does not aim to dictate how the phenomenon should take
place. Unfortunately most commentaries about Fitts's law seem to think
we should all abide by Fitts's law or leave this planet.

Anyway, in menus the entries are horizontally layered. The menus are
designed to open a submenu when the mouse in on the corresponding
entry. The problem happens if I try to follow a straight line from an
entry to an entry in a submenu. If the entry in the submenu appears
significantly lower in the submenu, the mouse will inevitably cross
over the next entry in the menu, which will open the submenu for the
next menu entry thereby forcing you to start again. Vertical motion
isn't unrestricted. It happens to me always when I quickly try to go
from the start menu to some application whose position I know thereby
trying to go straight from the button to that entry. If I am not
careful, I end up opening the submenu of the next menu entry in a
haste, then move left and up to find the right menu entry, then move
right _carefully_ to horizontally enter the correct submenu and then
rush down to the memorized spot. All in a wink of an eye and it has
become my second nature now. But every time this happens, I feel
slightly irritated. If menus are designed to take advantage of the
horizontal freedom of movement more, it would be much smooth. Deeper
nesting has the mentioned side effect. By the way, radial menu or
submenus that open in the middle or in the first entry should be
better. But apart from a few radial Firefox context menus, I have not
found any other implementation. I love radial menu.

That is all I had to say about start menus. Before finishing, I should
point out that even for the first group of self-organized users,
having someone or something to search is not totally out of the
question. Some form of data are too vast to organize (bookmarks and
directories wrt internet), some have no single way of organizing or
some even have no of meaningful grouping criterion. And even if you
arrange all your computer files in directories, emails in mail
folders, feeds in feed reader folders, websites in bookmark folders,
if you are to find this blog (containing "Fitts's Law" but unrelated
to beagle), then good luck!

I saved 80MB

2007-10-09T17:45:00.000-07:00

Its a common question in the database community where to store large binary
objects, commonly known as BLOBs (binary large objects). There are two
obvious options, store it in the filesystem or store it in a database. And by
twisting arms we get the third obvious approach, a mix of the two. There are
some studies in database academic community on the tradeoff.

Thinking carefully, the database and filesystem work in a similar way when
storing objects. Both store data in tree fashion, providing fast access to
any object. Both can add or delete object. And both suffers from
fragmentation.

The fragmentation is the key issue in my test case. So what is my test case.
It consists of lots and lots of small files and I mean, really small files
and some very large files. Just to give an example, I have a total of 20000
files. Of them about,
500 are above 4K
1000 2-4K
2000 1-2K
16000 less than 1K.

There is a reason for this extremely skewed distribution. The files are
textcaches in my home directory. The main contributing factors are about 8000
files (a mix of source code, tex, pdf and html and oh ... lots of pictures)
and 19000 emails (without spam). Now source codes are self caches i.e.
separate textcache is not stored for these files. Same with txt. Pictures and
other media has no text to cache. So the bulk of cached text comes from html,
pdf and office documents. Most of such documents in my home directory are
generally pretty large i.e. with a significant amount of textual content.
This is in sharp contrast to the behaviour for emails. The textual content
from the emails are stored. Also, unlike the html files on the disk, the HTML
parts in these emails are small, really small. Those contribute to the huge
number of extremely small files. If you are wondering how come I got 3000
absurdly small 20 byte files, I store the gzipped copy of the extracted text
in the textcache.

So now the question is how to best store data with such distribution. The
simplest thing is to use the filesystem. Filesystem are equipped with
sophisticated methods to avoid fragmentation. But with a fixed block size
(like 4K on my home directory partition), there is bound to be some
fragmentation when storing small files like above. On the other hand,
historically databases were concerned with storing small strings (with all
sorts of querying and update speedup; and they are pretty good at that).
Recently, most of the databases have added support for large binary objects.
There is also a small twist to this problem. The data is mostly static but
there are deletions (and sometimes modifications). The last important factor
is I am using the embedded sqlite database which stores all its data in a
single file; so in essence, what I get is a "fast retrieval and
update"-enabled huge tar file.

The prevailing folklore is that database is better for smaller objects and
filesystem for larger objects. What is small and what is large depends on the
nature of the problem at hand. There are some papers doing detailed analysis
of the tradeoff, and they conclude somewhat along the same lines. I found one
paper showing that adding and deleting lots of large objects over time causes
enough fragmentation in a database that it can lag behind a filesystem.

I am not much of an experimental person myself. With the above testcase, I did
not even think twice about what I should do. Well, for other reasons (like
heap fragmentation in my application, since reading from a database would
require me to store the entire data in memory) I was a bit biased towards the
filesystem approach. So I decided to take the middle path of a mix of
filesystem and database. Thanks to decent APIs, implementing the hybrid
approach did not turn out to be anything complicated. I store the first 4K in
a buffer and then if I get more data, move everything to a file otherwise
save everything in the database.

All the 19500 files with less than 4K size are now safely sleeping in the
database, which would have otherwise taken 4K each.

(* The title is not entirely correct. I did not have to waste 4K blocks for
each of 19500 files, but now my sqlite database is 26MB instead of something
much smaller. But I am confident, this is good for the mankind at large and
for the dolphins too.)

Tracker-ing looks fun

2007-09-13T04:56:00.000-07:00

Jamie checked in thunderbird support to Tracker today. How do I know ? Because I subscribe to svn-commits-list and my gmail filter separated him out when he was adding the beagle thunderbird backend (probably with some modifications, haven not checked the changeset) to tracker.

This is looking pretty cool. I also found out Tracker-0.6 was released with some extremely useful features and a damn good UI. The UI groups the documents in an intuitive way, shows more context information from matches, allows you to do tagging inplace (and has gradients, uh-on). The core daemon (is that what it is called ?) can scan evolution mails, and pidgin logs. Now it can also do index thunderbird mails. It lists XMP metadata support too - which is nice and something I never managed to finish in beagle.

I am pretty sure people will find it useful. I could not resist myself to admire it and share it with others. Well, I was told that tracker is set to become the default desktop search engine in ubuntu, so a lot of unhappy souls are about to become happy soon.

For me, I will continue using beagle. Why ? Because, it does what I need it to do. And if it does not do something, just remember it is written in C# :). Meanwhile, you make your own decision.

How is WebBeagle for a name ?

2007-09-09T17:34:00.000-07:00

(Or FireBeagle ?)

I always wanted to search beagle using a web-browser. My desktop searches are mostly infrequent but complicated. So if I am unable to find something I definitely need an index-based search tool (e.g. beagle) but on the other hand I don't want to keep an application open on my desktop for a long time (beagle does not consume significant memory for me, so it can keep running in the background). That was one reason why I wrote the kio slave for beagle (I have stopped liking it for a long time) and followed it with a kde deskbar like applet (which is nice but with limited functionality). Now that I am done with shameless advertisements, let me share with you yet another way to query beagle.

Today I wrote some ajaxy,xslt-ed webpages to allow users (read: me) to query beagle using a browser (err... firefox, konqueror-3.5.5 does not have XSLT processor). It uses the networkservice backend that can be used to query beagle over network (based on last year's Google Summer of Code projects).It uses internal knowledge of how queries are serialized, lists all the information in a boring way and does not show snippets (yet). But it works and was reasonably fast in displaying 42 results. Besides the boring UI, allowing browsers to access services always opens up some security hole, so it is disabled by default. If you so desire, use it but at your own risk (check the commit log for how to turn this feature on).

I do hope to get this feature properly implemented. The things need to be done are:

Fix the network-backend (it is sufferring from some crashes).
Probably related to the network backend as well, some kind of search authorization is needed.
Use CSS + javascript for the results page to group/sort the results, and make them look decent.
Get snippets and display them. I am thinking of retrieving snippets only on demand. Somehow the name and the location of files or the sender and the subject of emails help me more in filtering out search results than snippets. Which is quite unlike how I use web search engines.
Figure out a way to use the C# or libbeagle API to create the xml request messages. Currently they are hardwired. If the solution turns out to be too complicated, it might not be a bad thing to leave the format hardcoded as it is now.
More cosmetic, separate command line and configuration options for this feature.
This is more related to the network query implementation; figure out how to use the QueryDomain thing meaningfully. The results from some backends only make sense on the same machine and even for some of them, it is tricky to open the applications just from the URI itself. So does it make sense to show e.g. evolution mail hits in the browser ? Does it make sense to return gaim (pidgin) hits when queried over network ?
How does the browser behave (read: choke) when it receives 1000 results (100 results each from 10 backends)? (i.e. the DOMParser has to parse a huge string and form a huge DOM of 1000 Hit nodes. You see why I dont like to get all the snippets beforehand :)?

Hey Firefox, Beagle This ... NOW!

2007-08-09T20:51:00.001-07:00

Consider this scenario: B was writing an article when he remembered that he read a relevant blog some weeks ago but did not bookmark it due to his lack of foresight (stupid B). What to do ? He could possibly run a grep in his Firefox cache folder hoping that the cache would still be there or scan his FF history for a familiar looking URL. Anything better ?
If B was using the FF beagle extension, he could find the blog instantly

Really! Then I will pose this requirement. B has been using the new FF beagle extension for some time now. But he does not want to index all his browsing history. Sometimes he just wants to browse off the record. Maybe after a while he wants to index the pages he visits. Pretty random ... but hey, that's B's nature.
No problem. The new extension puts a small status icon, which can be used to disable/enable the automatic index; really, at B's own sweet will.

That was easy. How about this one, a bit harder. Someone told B about FF beagle extension, and B started using it. But this time he needs a PDF paper he found online. He does remember that he read the paper and he must have read it in the browser itself. Now what ?
If B was using the new FF beagle extension, he could even search for embedded documents. Embedded content is not indexed automatically (as of now) but can be indexed as required using the status icon menu.

Hmm... But what if B visits a page with auto-indexing disabled and then decides that he better index the current page. Now what ? Does he have to enable the auto-indexing and then reload the page. Thats stupid!
Heh. You gotta be kidding. Of course he can use the status icon menu or the page context menu to index the current page.

If B wants to index an image, or take this one, sees a link to something (PDF, image, java source file, mp3 ...) and wants to index it without loading it explicitly ?
The context menu provides options to index an image and index a link; in both the cases the file is retrieved in the background and indexed. All filetypes that beagle can filter, can be indexed using the index this link feature.

This new extension seems to uber-useful. I guess I err... B should really start using it. Can B save notes while indexing on demand i.e. if he is indexing a link he might want to index a few extra keywords (for querying or for reference) ?
Yeah! B gets that too. If B checks the corresponding preference entry, the extension will prompt for extra keywords to save and index when B explicitly asks it to index something.

I will be good and share this new beagle extension with everybody, written by Tao as part of Google Summer of Code 2007. Soon to be found in beagle trunk.

Mr Beagle, you are on the air

2007-07-06T15:16:00.000-07:00

Back in India, I did one good thing. Finished adding Alexis Christoforides' project of Google Summer of Code 2006 Beagle over the network. Its a bit untested with the recent versions of beagle and I didn't read the code completely, so I left it in a separate branch for the time being.

What can you do with this wonderful project ?
- Start beagled on one machine
- Add that machine (with port 4000) to the list of neighbourhood nodes via beagle-config on a second machine in the network
- Remotely query the beagled on the first machine from the second machine

The SoC project was a complete one, with GUI support. However, I only added the basic infrastructure and enabled the cli tool beagle-query to query remotely. I am not sure if querying for emails remotely makes sense, how will the results be opened ? As of now, only the filesystem backend is enabled to send query over the network.

Ideally, beagle should automatically find out other networked beagled in the network. Don't despair, there was another SoC project in 2006, Avahi Integration by Kyle Ambroff doing exactly that. Sadly I am too preoccupied to work on integrating the Avahi work. But with that and the GUI parts of Network Beagle, beagle would be useful in more ways than ever.

Silence of the Dog

2007-05-17T19:23:00.001-07:00

Lately beagle releases have slowed down quite a bit; there were a few bug-fix 0.2.16.x release and another 0.2.17 bug-fix release (it was supposed to 0.2.16.4, but the changelog was too large for a point release). The underlying goal is to get ready for 0.3.0; svn trunk is changing so rapidly these days that it is difficult to isolate the simple ones and make them into a 0.2.x release. On the other hand, the changes are too major to be put into a 0.2.x release (they would also need extensive real life testing).

Recently I moved beagle to use taglib-sharp for filtering music files from entagged-sharp. I was told entagged is no more actively maintained and taglib is definitely seeing a lot of rapid development. My timing was not quite right, the 4th March news "Entagged is unmaintained" is followed by the 28th March news "Entagged is maintained". I came to know about it only after I made the transition. Too late! On the plus side, taglib# has support for larger number of formats and is being used by Muine and Banshee, so expect sharing of taglib-sharp libraries. Unfortunately, there are no taglib-sharp packages out there yet (there is a proposal for a debian package), so all the mono apps are currently including taglib-sharp by its source. We too initially source included it, then removed it and instead linked against the package. But if there are no packages for the major distributions, it might make sense to source include it; compiling Beagle is pretty demanding anyway.

In other news, I used the extremely handy heap-shot to identify that instances of IndexReader were not being GC-ed even long after the corresponding method ended. Explicitly setting them to null immediately freed them. I suspect some thread local storage magic happening behind my back. Note to self, set IndexReaders to null immediately after they are closed. Did I say heap-shot is amazing ?!

There are several more improvements to the speed and memory performance of IndexHelper and BuildIndex. One notable feature I added was to reduce re-indexing of files which could not be filtered before. Due to the inherent distributed nature of beagle indexing, the crawler is always separated from the indexer. So if the crawler finds some file which was not filtered before, it has to re-submit it to the indexer. Who knows! There might be a suitable filter now. The downside was that a lot of files were being repeatedly re-tried by the indexer, slowing down the whole process. I decided to store the files containing the filters and their last modified times in a filterver.dat (akin to mozilla pluginreg.dat) and if the filters were not changed since last run, assume that there is no newer filter. Fair guess I would say.

Beagle knew how to index email attachments for quite some time; some months ago it also got the ability to index archives. However all along this was done by extracting the included files to a temporary file and then indexing it. This was done primarily because of the way included content (aka child indexables) were handled and also due to the fact that some of the filters only worked on physical files and not streams. This whole temporary file business never pleased me, there were race conditions which could leave undeleted temporary files in the system, even small included files had to be written to disk and further, extracting the contents of an archive to index it defeated the whole purpose of archiving it. Last week, I added the infrastructure to allow indexing of archives and email attachments without extracting them, if the filter permits of course. The infrastrusture is there, the archive and email filters should be modified to take advantage of this.

Finally, one feature I personally would like to see in 0.3 is support for XMP sidecars. XMP sidecars allow users to add a separate file.ext.xmp file containing arbitrary metadata (but in the XMP format) about file.ext. Really extensible solution for metadata. The main part of the code is in svn trunk; it still does not support renaming or deleting xmp files. Hopefully it will be finished in time.

This will probably be my last post before my annual break to the land of mangoes (" fruit of the gods"). Sadly, I have (knowingly) only tasted about a dozen varieties of mangoes, out of over 300. I will definitely try to increment the number this time. Next post, July.

Upgrade to Spring

2007-05-03T11:58:00.000-07:00

Yesterday in a fit of mind I decided to upgrade to Mandriva latest release.
The steps included
- backing up .kde, .kderc, .qt, .gtk* and .local
- logging out of kde
- setting up a mirror as a distribution source for urpmi (mandriva is offering
non-free e.g. sun-java in its free source these days, but I still need the
PLF sources for codecs, BCI enabled freetype and fontconfig and a few other
things)
- # urpmi urpmi
- # urpmi <bunch of> kernels
- # urpmi --auto-select
and then selecting the ones I would like to upgrade from them

Soon I was running Mandriva 2007.1 Spring (Free). Yay! Its beautiful. The
Ia_ora theme and other Mandriva artwork is gorgeous. I legally own a XP cd,
from which I extensively use Verdana (for text) and Tahoma (for widgets).
They look wonderful as always with plf freetype (w/ hinting). I like to use
large fonts, enough to be readable 4 ft afar but somehow the deja or
bitstream fonts have a weird fuzziness in the curves of 's' and 'o's. I
cleared the settings of a test user account and a default new account looks
quite good (apart from the kbfx-styled mandriva menu).

KDE was upgraded to 3.5.6; I was actually using a few kde-3.5.6 packages from
cooker so there was no huge surprise. I was worried that mandriva would mess
up some of my settings when I log in for the first time as my normal user but
thankfully that did not happen. The system feels faster, konsole definitely
starts faster than before. Overall, I am extremely pleased with 2007.1; I
wish them all the best.
http://wiki.mandriva.com/en/Releases/Mandriva/2007.1/Tour

Some minor annoyances:
- mandriva kernel (based on 2.6.17) still has the weird cpufreq bug where
scaling_max_freq is same as scaling_min_freq (thus rendering all the
governors useless). It is probably the same problem described in

http://www.mail-archive.com/linux-acpi@vger.kernel.org/msg04484.html
- Suspend to RAM is broken in mm kernel (Mandriva has moved to pm-utils and it
works like a charm w/ the default kernel)
- tmb kernel has some problems of high CPU usage when copying files and broken
resume from s2ram. I still need to test the other tmb versions. I really like
the tmb kernel improvements.

Theory Guy...

2007-04-20T12:05:00.000-07:00

This is not really techie but has a somewhat tech-vs-theory war flavor. The following list came out of today's departmental Friday lunch. Top 10 "ways to tell a theory person from a systems person":

10. One gets job and another does not.

09. One likes 10000n^2 better than n^3.

08. Space-time is important outside Star-trek.

07. In therory they are same, but in practice they are not.

06. P is not equal to NP divided by N.

05. n^10 is efficient.

04. SAT (Famous NP complete problem also known as Satisfiability) is most of the times easy to compute.

03. (Specific to our department) One can be only be found across the street.

02. Its pronounced "Lee-nux" (not "Lai-nux")

01. Computers do more than just email ?!

TinyBeagle or a Lucene Example

2007-04-04T14:56:00.000-07:00

Recently I read this interesting comment in an OSNews article. It tried to briefly summarize what beagle is. I take users' comments very seriously and this person seem to know some internal of beagle, so I thought maybe he is true (modulo some factual errors). Maybe the only new thing in beagle is the crawler, the GUI and the scheduler; its mostly little C# glue code tying up a few third party apps.

So, I wrote down a small Lucene.Net based file indexer and query program. You index by
mono LuceneLocate.exe /path/to/index/dir index /directory/to/index
and query by
mono LuceneLocate.exe /path/to/index/dir query query_term
Pretty simple program, 85 lines of actual code. Incredibly fast performance. Using external program ('cat') to index files in a directory (recursively), it indexes 180 files in 0.06 seconds. Query returning 44 results took 0.0015 seconds . Takes 24 MB virtual, 5.3 MB RSS-Shared. No GUI yet. I could have added a scheduler to pause for 10 seconds after every 10 files (5 more lines). This Lucene.Net based crawler and indexer beats beagle in performance but nowhere close to beagle.

Maybe beagle is not a lucene-powered locate. After all, to err is human.

And PLTC means ...

2007-03-01T13:23:00.000-08:00

I just now saw the programmer's personality test webpage in Digg and decided to give it a curious shot. And ... my programmer personality type is PLTC.

(Update: The link to my personality doesn't work. I thought they have one static page for each possible combination. I am probably wrong, let me add the full text here)

You're a Planner.
You may be slow, but you'll usually find the best solution. If something's worth doing, it's worth doing right.

You like coding at a Low level.
You're from the old school of programming and believe that you should have an intimate relationship with the computer. You don't mind juggling registers around and spending hours getting a 5% performance increase in an algorithm.

You work best in a Team.
A good group is better than the sum of it's parts. The only thing better than a genius programmer is a cohesive group of genius programmers.

This one I like the best.
You are a Conservative programmer.
The less code you write, the less chance there is of it containing a bug. You write short and to the point code that gets the job done efficiently.

OMG. That's totally like me. Or whatever!

kBeagleBar is alive!

2007-02-19T19:11:00.000-08:00

I did not guess that people still use kBeagleBar, there are packages built for it and there was even an article about it and other KDE beagle search tools. Nice!

Faceless Bugs and Advanced Users

2007-02-19T09:20:00.000-08:00

These are really two very different topics but they came to my mind while reading about Linus' Gnome patches and bugs.

The first is about creating a new account when I need to report a bug or submit a patch for some software. Most of the projects prefer attaching to bugzilla or they send it to their member only mailing list. I am extremely reluctant to create new accounts, so I have created bugzilla and mailing list accounts for KDE and Gnome. That covers a lot of ground. But still now and then I face a need to send something to somewhere else and bam! Sign up for an account sir! There is definitely merit in this approach, since otherwise bugzilla and mailing lists would be flooded with spam. But it definitely keeps me from submitting patches or commenting on something due my lack of interest in new accounts. Last week, a friend of mine (the inventor of Sperner's Game) was trying to install Kubuntu in his brand new Lenovo T60 when he spotted some typos in the installation windows. He was ready and willing to file a bug in Kubuntu and was told to create a new account for kubuntu bugzilla. As always, he was supposed to get a confirmation email.

The email came 12 hours later and I do not know if the bug was ever filed! Even if the email was prompt, the desire to report a bug has to be high enough to cross these technical potential barriers. *sigh*

This week I made extensive addition to beagle query syntax. There is an open bug in bugzilla asking for a visual way to add these advanced query expressions in beagle-search. I was thinking how best to achieve that; it is not easy to capture the power of beagle query expressions in a gui. I found the answer while reading some posts in desktop-architect mailing list about Linus' patch. There is nothing like an expert user or a novice user. Users always try to act as if they are smart and take the path of the expert user. Presenting different set of options for these different class of users does not work in practice.

In a similar style, there is no need for a GUI for advanced query expressions. Novice users i.e. users who will simply enter search terms will never know what a full boolean query expression does (with those OR and excluded expressions). On the other hand, expert users who know how to deal with the boolean expressions, the different keywords to do property search and other advanced syntax can anyway write it by hand. In fact, it is much easier for them to write it by hand than to do it visually. In this matter, I like the approach taken by
Google. I think I will push towards a simpler advanced search UI for beagle-search and Kerry, some simple choices like choosing type of file, extension, date range etc. Write the query by hand if you need that extra ounce.

beagle memory usage

2007-02-11T19:02:00.000-08:00

Setup: Fresh run of beagled with only the kmail backend. IndexInfo report about 13700 items i.e. mails and indexed attachments. Beagle version is post 0.2.16, so that includes the individual items in the archive attachments as well. I started beagled as exercise_the_dog, indexing finished within an hour and this is the state after indexing is over.


VIRT      RES   SHR     COMMAND
--------+------+------+--------------------
167m    55m  11m     mozilla-firefox
248m    29m  2856    X
137m    20m  15m     amarokapp
72812   19m  6884    beagled-helper
89560   18m  13m     kmail
35816   15m  11m     konsole
42780   15m  14m     konqueror
49088   12m  5860    beagled
40320   11m  9524    kdesktop
42004   11m  9.9m    basket
32620   10m  9588    kmix
43896   9252 6220    kicker
37904   5884 3908    kded
34560   5600 2680    net_applet

Remember the rule: an approximate idea of the memory usage is given by RES-SHR.

beagle:Eat less, talk less be smart

2007-02-08T05:27:00.001-08:00

Yesterday, beagle 0.2.16 was released. A couple of weeks back, we released 0.2.15 but I did not write about it. 0.2.15 came with a lot of performance and memory improvements, new backends, new features, lots of important changes . In the process, it also broke a few things. Those were fixed and 0.2.16 is a purely bugfix release for 0.2.15. I am considering 0.2.16 the best ever beagle release. Incidentally, 0.2.13+ releases somehow or the other had some nasty problems.

Combining 0.2.15 and 0.2.16, these are the major improvements:

* Very important, the looping bug is fixed. I would even like to claim, fixed forever. I happened to find some important clue while scanning the logs and other information provided by some of our very friendly and helpful users. Eventually our 3 year old database schema was found to be incorrect. Joe finally cleared the mess. Thanks Brian and Rick! This also means an end to the "log file filling hard disk" or "beagle indexing even after a week" type problems.

* Beagle uses some external tools to filter files e.g. pdfinfo, pdftotext, mplayer yada yada. These programs are well written and almost always work. Except some very malformed or wrongly detected mimetype file is sent to them and they go berseck taking up insane amout of memory or CPU time. Since the early release, we used to maintain that there is no way we can control the external processes. After all, we just use 'em. Joe finally put an end to that excuse by using some smart rlimit tricks to limit the resources used by these external processes. We still cannot control how mplayer might behave if given an word doc file, but if it behaves badly it will be killed before too long.

* Indexing data is a strenuous job. Think about all those heavy applications which process or generate these files. But people want indexing to be as silent as possible. There are frequently recommendations that beagle should use high nice, low system priority. low IO priority etc means to be as unobtrusive as possible. The fact is, beagle already does that. However, now we even go one more step by using SCHED_BATCH scheduler policy.

There are other side improvements too, RTF filter is new. The current one is based on the legendary RTF parser by Paul Dubois. Image filters are almost new; we now have Konversation (KDE IRC client) and KOrganizer (KDE tasks and eve nts scheduler) backends. By the way, soon after 0.2.16 was released, Opera webhistory backend was added to trunk. You can just drop the binary from here into your 0.2.16 /usr/lib/beagle/Backends folder and start using it, err... trying it. I do not know how complete it is.

I would like to end by thanking the excellent user base that beagle has developed. Without them, it would not be possible to fix a whole lot of these problems. Beagle would not be what it is today without them.

Seekable LineReader

2007-01-11T15:17:00.000-08:00

Recently I need a way in StreamReader to get the position of a line which can be stored and later skip to that line directly. C# StreamReader API does not have any way of doing this, except calling Read() repeatedly and then doing the processing yourself. Which is clumsy. Note that, StreamReader.BaseStream.Position might be wrong due to underlying bufferring. I thought this should be a fairly common requirement and indeed, many people have same question on google groups or other forums. No good answer though. One reason is, such a thing does not really make sense for arbitrary Streams, since it might not be possible to seek in them.

I needed such a thing desparately, so I created an interface:

namespace System.IO {

 // A linereader interface
 public interface LineReader {

   // Returns a position marker, which can be used to navigate the lines.
   // Some implementations may only allow moving in the forward direction.
   // Might be different from line number or file offset.
   // Should only be used for traversal.
   long Position {
     get;
     set;
   }

   // Reads and returns the next line, null if EOF
   string ReadLine ();

   // Reads the next line and returns a stringbuilder containing the line
   // The StringBuilder returned could be the same one used while reading,
   // so it should not be modified and its content might change when readline
   // is next called.
   // This is the most worst horriblest API I ever designed, for sake of speed
   // And thats why this should not be a public API.
   StringBuilder ReadLineAsStringBuilder ();

   // Skips the next line, return true if successful
   bool SkipLine ();

   // Skips required number of lines; returns actual number of lines skipped
   long SkipLines (long n);

   // Close the reader
   void Close ();
 }
}

Beagle source contains the interface and several implementations.

Fasten your seatbelts; we are ready to ship

2006-12-31T14:32:00.001-08:00

There was a discussion going on in beagle mailing list sometime ago where I made a comment that I dont think beagle is newbie ready i.e. plug-and-play yet. Beagle fans did not understand my comment and people replied why they think I am wrong. I would be glad to be proved wrong; but all of their arguments were how they were using beagle since version x.y, how beagle was shipped and enabled by default since last z releases of some distribution and how someone is able to install beagle in a large enterprise. Duh! None of these prove that beagle is newbie ready. All they do is show that beagle works and even I know that. 0/100.

I have the feeling that some of the beagle devs and followers live in the garden of Eden surrounded by a high wall of reality. Sometimes they should go out in the streets, check the bugzilla of other distros, go through user blogs (which mostly contain complaints about how beagle does not work and how to disable it), and visit some user forums where a lot of questions are how to disable beagle from starting at startup. These are laborious jobs and not pleasing. A lot of them contain flames and invalid reasons. But almost always they are started by someone who found beagle causing trouble.

Here are some links which can make your task easy:

I sometimes make the rounds and all I see are I make a point of uninstalling beagle on all my machines and The first thing I did after ... was to uninstall beagle and now my machine is happy. Silly men, how can they not like the doggy!

Subversion arrives. Finally!

2006-12-30T19:49:00.001-08:00

KDE uses SVN for their source code management. GNOME used to use CVS till yesterday, which means beagle too was managed with CVS. I do not know the technical details. but time a again we did face technical problems with CVS. I was mostly told that life would be easier with SVN and lo behold! GNOME has switched to Subversion as its SCM (actually, still switching).

The last time this was tried by the awesome GNOME guys, they later found a glitch and had to cancel the migration. As a result I lost a commit that I made within hours of SVN migration. This time I will play safe and watch it for a few days before committing anything. If everything works out, life should be easier. Joe already cleaned up quite a bit of the unused files and directories, renamed the Evo-mail backend correctly and updated the links et al. A New Year with a clean, new repo. Sweet.

PS: There is one downside though. Joe (and others too) would like to use the SVN commit messages for creating Changelog files during creating a tarball. Which basically means others cannot observe the Changelog file between releases to figure out what was changed (neither I nor Joe updated the Changelog while committing, so this is a lame excuse). The real trouble is now I cannot write any lame jokes in my commit messages. Life would be serious now. Boo hoo.

My time with the doggie

2006-12-17T15:13:00.001-08:00

Today I read about the Ohloh project (http://ohloh.net) and added my favourite project beagle (http://beagle-project.org) to it. I was curious what it actually does.

It took them nearly 4 hours to download and analyze the source code. But it was worth the wait. It showed some interesting statistics, like 122,885 LOC codebase, 82 direct contributors (committing in CVS) and 13 of them in last 12 months.

Just for a light comparison, Firefox has a codebase of 157,207 LOC, Amarok has 169,288 LOC and (take this) PHP 6.0 has 599,805 LOC.

It was also amusing to see my share in the project: http://ohloh.net/projects/3826/contributors/21154

beagle 0.2.14

2006-12-14T15:06:00.000-08:00

Joe announced the release of beagle 0.1.4 few hours ago. Its a fascinating and shining new release containing exciting new features, lots of memory/speed optimizations and many bug-fixes as well. Here are the major ones which are readily visible:

Indexes tar, gzipped-tar, bzipped-tar, gzipped and bzipped files, in the filesystem as well as in email attachments. The results show you the exact file in the archive that matched the query.
Do some smart tokenizing to allow matching 001234 to a query of 1234 and better matching of file names. No more missing files.
Beagle can find and extract data itself using its dozen or more backends. But sometimes its better for other applications to send data to beagle for indexing. Beagle had the infrastructure to act as a search/indexing service provider. The release contains an example C code to show how to do that; its pretty simple actually. Obviously python can also be used.
Some cool signal mechanism which help to figure out what file in being currently indexed and for how long. This will be helpful if you feel beagle is taking ages to index some file.
Use Xdg autostart mechanism to auto-start beagle. KDE4 will also implement xdg autostart mechanism. One more step towards being DE agonistic.
The indexing information now explicitly mentions if the initial indexing is in progress. Also clients now have the option of being notified when the initial indexing ends.
Lots of memory fixes. bhale just mentioned in the irc channel holy crap, startup RSS for beagled is 15m... beagled is below nautilus in mem usage... im not believing my eyes :) Thank you for your myth on how beagle is a bloatware.
API and beage-search support to know the total number of documents that matched any query. Not the superficial imposed limit of 100 documents.

Go, get it!

License to hack-debug-release

2006-12-03T14:12:00.001-08:00

I start with the disclaimer that I am not a lawyer. I am also not very careful in reading the EULAs (End User License Agreement - a short acronym of a long term usually describing an even longer gibberish english text). When I release my first piece code (was it JBabel or mGet ?), I put them in GPL (possibly v1). Of course I didnot know what was I doing - I was merely completing a formality during the fascinating experience of sharing one's program. With time, and during development of Beagle, I came to know about the various licenses. It was about this time that the GPLv3 debate started. Also, I started releasing verious kde-beagle softwares which required me to figure out the correct licenses for them.

Liceneses, I feel, serve two purpose, demand credit where its due and specify the amount of responsibility.

Authors of some projects just want to release code without claiming any credit. I did so too. Only to be pointed out by someone that I have to actually specify the terms for others to use my code. Since I dont want to be credit for the code, its natural not to held responsible for its damages either. That needs to be specified too. One can either cook up one's own license or just pick one from the sea of licenses (http://www.opensource.org/licenses/) . The license I found to do just that is the MIT license.

MIT (X11) License: Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, ...

The license is itself not copyrighted, so it can be modified if needed. A rather free-to-do license, except for this clause following the above text

... subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

Wikipedia tells me that the latter part is necessary for the copyright laws of US and other countries. The credit part is followed by the author responsibility part

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT...

Often, people use this clause to explicitly deny responsibility

Except as contained in this notice, the name(s) of the above copyright holders shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Software without prior written authorization.

I havent seen projects where the last clause wasnt present, so I assume it is not a necessary clause. I didnot use it myself when I released yaBi under MIT license. The core parts of Beagle and Mono are released under this license.

My other project, kBeagleBar uses the fancy LGPLv2. I had to use LGPLv2 because I was using some source code from kde libraries. Note that, LGPL allowed me to use the library without any restriction, but I copied some code from a library source file - so I had to use LGPL. Similarly, I release kontrol as (distributed as a part of kbeaglebar) which uses the code from GPLed kerry, so its under GPL too. Also GPL and LGPL are copyrighted by FSF. But then again, I wouldnt dare to modify them myself.

I am looking forward to GPLv3. No, not because I have any particular reason to like it. Because I hope to read it fully, understand it and read the various articles and blogs about it to understand the issues of free software licensing. Its always good to know where my rights end and start.

Even after all this, it will take me some more years to actually read EULAs. They are frightfully convoluted to confuse my little grey cells.

Bigger than life icon

2006-11-26T09:03:00.001-08:00

Icons are small pieces of art I encounter in my everyday life. They are like
apparels, you see them in all forms in every moment of life. They serve
different purpose, keep you warm, keep you cool, keep you comfortable
whatever but all them try to look good (some even make it their prime
agenda). In human-computer interaction, icons are large, easy targets that
serve as better visual cues than textual messages that needs to be parsed and
pattern matched. A good icon make that experience fun and enjoying.

I started with the the usual KDE crystal icon theme, and then tried different
icon sets. The tango icon theme is good, but like the usual gnome colours,
they have a very low-saturated colours which look a bit drab and boring on my
desktop. I am not looking forward to extremely gorgeous, highly saturated and
colourful icons; my desktop is not a colour palette, but the art forms I will
stare at a million times over months better be pleasant to look at. kde-looks
has quite a few icon theme, and a lot of them are mixed icon theme. For a
while, I tried KDE Crystal Diamond icons. It has good icons, except some of
the icons are very light coloured, white based and its hard to make out the
icon contents. Most of the filetype icons and the media-play (amarok)
play/pause icons are like that.

I have heard a lot about Oxygen icons; unfortunately their license does not
allow anyone to package them. Of course, I could download the svg icons
directly from their svn and convert them to an icon theme locally. I need to
install inkscape for that. Interesting enough, I found a forum post in
sabayonlinux about using oxygen icons in current kde desktop. The user gives
out the details of how to make an icon theme and later even provides a tar.gz
package to install oxygen icons. Oxygen is work in progress, so that user
replaced the missing icons by other icons. I think he violated their license
but I have to double check. Anyway, I installed that icon them and its really
good. Attention has been paid to every detail. Icons are 3d and 2d exactly
according to their need, those that look good with shadows have them, the
colours look just ok. All in all, I am pretty satisfied. I am looking forward
to manually checking out the oxygen icons and making a theme out of it. That
way, I will get the icons from oxygen as they are checked in. Good work,
guys.

Mono.FUSE Filesystem for Digikam Tags

2006-09-23T23:07:00.000-07:00

FUSE is cool anyway. And writing FUSE filesystems in C# is even cooler. Adjectives aside, I wrote a FUSE-DigikamFS filesystem for browsing Digikam tags. Its much easier to browse the images organized by tags using konqueror or kuickshow. This way I can also share the tags mount-point by kpf (or some other HTTP server) allowing others to browse my images arranged by both folders (Albums) and tags.

A sample session,


[debajyoti@dbera Tags]$ pwd
/home/debajyoti/Tags
[debajyoti@dbera Tags]$ ls
Artistic/  Favourites/  Fireworks/  Food/
Nature/  People/  Places/  QuickCheck/  Season/
[debajyoti@dbera Tags]$ ls Artistic/
Black-White/  Mood/  Motion/  pa020099.jpg@
pa020102.jpg@  Perspective/  Shadow/

As you see, I use symlinks to overlay the images from tag filesystem. Works pretty cool.

Ed-ed

2006-07-24T12:04:00.000-07:00

Who knew that 'vi' was based on the legendary editor 'ed'. Or, inspired by. Being a die-hard vi-fanboy, it took me no time to get used to ed, at least being able to compose/edit/save simple documents. An ed-documentation page at BUCS provided me with a quick howto listing (all) the important commands. Actually, all the commands, since there are only a few of them. Unlike vi, where you have to know thousands of commands. Its wonderful how much simpler life becomes if we focus on functionality alone without caring for presentation.
If you are comfortable with vi (or even with vim or with gvim but dont use the menu too much), I suggest you to give ed a try. You'll love it.

(From the documentation page) An ed-quickcard:

All the standard ed commands are listed below, together with a brief description of their function. Those commands which may be given with line addresses are shown with the default values of the addresses in the first column. For example, 1,$ for w means that the write command may be given with one or two line addresses to specify a particular line or a range of lines to be written to a file, and that if no address is given the default address is 1,$ (ie, all the lines in the buffer are written out). The default address . represents the current line and $ represents the last line in the buffer.


The Input Commands

Default Address  Command  Function
.                a text   Append input text after addressed
                        line.The last line input becomes the
                        current line.
.,.              c text   Replace addressed lines with input
                        text. The last line input becomes the
                        current line.
.                i text   Insert input text before the addressed
                        line. The last line input becomes the
                        current line.

The Edit Commands

Default Address   Command   Function
.,.               d         Delete the addressed lines from
                          the buffer. The line after the
                          last deleted line becomes the
                          current line.
                e file    Delete the buffer contents then
                          read file into the buffer. The
                          last line read in becomes the
                          current line.
                E file    As for e but no warning is given
                          if a modified buffer has not been
                          written out.
                f file    Print current remembered filename
                          if file not specified, otherwise
                          set it to file.
1,$               g/R/cmds  Perform cmds on all addressed lines
                          matching regular expression R. Last
                          line in which a match was found
                          becomes the current line.
.,.+1             j         Join together all the addressed
                          lines. The resulting line becomes
                          the current line.
.                 kx        Mark the addressed line with the
                          single lower case character name x.
                          The addressed line becomes the
                          current line.
.,.               l         List the addressed lines showing
                          non-printing characters and folding
                          long lines. The last line listed
                          becomes the current line.
.,.               mA        Move addressed lines to follow line
                          whose address is A. The last line
                          moved becomes the current line.
.,.               p         Print the addressed lines. The last
                          line displayed becomes current line.
                q         Exit from the editor.
                Q         Exit from the editor with no warning
                          if a modified buffer has not been
                          written out.
$                 r file    Read file into the buffer after
                          addressed line. The last line read in
                          becomes the current line.
.,.               s/R/S/{g} Substitute string S for the regular
                          expression R in the addressed lines.
                          If g specified substitution is made
                          globally throughout addressed lines.
.,.               tA        Copy the addressed lines to follow
                          line whose address is A. Last line of
                          the copy becomes the current line.
.,.               u         Undo the effect of the previous
                          substitute command. The current line is
                          reset to its value before that command.
1,$               v/R/cmds  Perform cmds on all addressed lines
                          not matching regular expression R.
1,$               w file    Write the addressed lines into named
                          file. The current line is not reset.
1,$               W file    Append the addressed lines to the
                          named file. Current line not reset.
                x         Decrypt or encrypt the text according
                          to an input key. The current line is
                          not reset.
$                 =         Print the line number of the addressed
                          line The current line is not reset.
                |cmnd     Pass cmnd to the UNIX shell to be
                          executed. Current line is not reset.
                A         Where A is one of the legal address
                          forms listed above: locate addressed
                          line and display it. The addressed line
                          becomes current line.
.+1               newline   Print the next line. Addressed line
                /linefeed becomes the current line.

Obligatory jokes link (not a PJ).