kBeagleBar is alive!

I did not guess that people still use kBeagleBar, there are packages built for it and there was even an article about it and other KDE beagle search tools. Nice!

Faceless Bugs and Advanced Users

These are really two very different topics but they came to my mind while reading about Linus' Gnome patches and bugs.

The first is about creating a new account when I need to report a bug or submit a patch for some software. Most of the projects prefer attaching to bugzilla or they send it to their member only mailing list. I am extremely reluctant to create new accounts, so I have created bugzilla and mailing list accounts for KDE and Gnome. That covers a lot of ground. But still now and then I face a need to send something to somewhere else and bam! Sign up for an account sir! There is definitely merit in this approach, since otherwise bugzilla and mailing lists would be flooded with spam. But it definitely keeps me from submitting patches or commenting on something due my lack of interest in new accounts. Last week, a friend of mine (the inventor of Sperner's Game) was trying to install Kubuntu in his brand new Lenovo T60 when he spotted some typos in the installation windows. He was ready and willing to file a bug in Kubuntu and was told to create a new account for kubuntu bugzilla. As always, he was supposed to get a confirmation email.

The email came 12 hours later and I do not know if the bug was ever filed! Even if the email was prompt, the desire to report a bug has to be high enough to cross these technical potential barriers. *sigh*

This week I made extensive addition to beagle query syntax. There is an open bug in bugzilla asking for a visual way to add these advanced query expressions in beagle-search. I was thinking how best to achieve that; it is not easy to capture the power of beagle query expressions in a gui. I found the answer while reading some posts in desktop-architect mailing list about Linus' patch. There is nothing like an expert user or a novice user. Users always try to act as if they are smart and take the path of the expert user. Presenting different set of options for these different class of users does not work in practice.

In a similar style, there is no need for a GUI for advanced query expressions. Novice users i.e. users who will simply enter search terms will never know what a full boolean query expression does (with those OR and excluded expressions). On the other hand, expert users who know how to deal with the boolean expressions, the different keywords to do property search and other advanced syntax can anyway write it by hand. In fact, it is much easier for them to write it by hand than to do it visually. In this matter, I like the approach taken by
Google. I think I will push towards a simpler advanced search UI for beagle-search and Kerry, some simple choices like choosing type of file, extension, date range etc. Write the query by hand if you need that extra ounce.

beagle memory usage

Setup: Fresh run of beagled with only the kmail backend. IndexInfo report about 13700 items i.e. mails and indexed attachments. Beagle version is post 0.2.16, so that includes the individual items in the archive attachments as well. I started beagled as exercise_the_dog, indexing finished within an hour and this is the state after indexing is over.

VIRT      RES   SHR     COMMAND
--------+------+------+--------------------
167m    55m  11m     mozilla-firefox
248m    29m  2856    X
137m    20m  15m     amarokapp
72812   19m  6884    beagled-helper
89560   18m  13m     kmail
35816   15m  11m     konsole
42780   15m  14m     konqueror
49088   12m  5860    beagled
40320   11m  9524    kdesktop
42004   11m  9.9m    basket
32620   10m  9588    kmix
43896   9252 6220    kicker
37904   5884 3908    kded
34560   5600 2680    net_applet

Remember the rule: an approximate idea of the memory usage is given by RES-SHR.

beagle:Eat less, talk less be smart

Yesterday, beagle 0.2.16 was released. A couple of weeks back, we released 0.2.15 but I did not write about it. 0.2.15 came with a lot of performance and memory improvements, new backends, new features, lots of important changes . In the process, it also broke a few things. Those were fixed and 0.2.16 is a purely bugfix release for 0.2.15. I am considering 0.2.16 the best ever beagle release. Incidentally, 0.2.13+ releases somehow or the other had some nasty problems.

Combining 0.2.15 and 0.2.16, these are the major improvements:

* Very important, the looping bug is fixed. I would even like to claim, fixed forever. I happened to find some important clue while scanning the logs and other information provided by some of our very friendly and helpful users. Eventually our 3 year old database schema was found to be incorrect. Joe finally cleared the mess. Thanks Brian and Rick! This also means an end to the "log file filling hard disk" or "beagle indexing even after a week" type problems.

* Beagle uses some external tools to filter files e.g. pdfinfo, pdftotext, mplayer yada yada. These programs are well written and almost always work. Except some very malformed or wrongly detected mimetype file is sent to them and they go berseck taking up insane amout of memory or CPU time. Since the early release, we used to maintain that there is no way we can control the external processes. After all, we just use 'em. Joe finally put an end to that excuse by using some smart rlimit tricks to limit the resources used by these external processes. We still cannot control how mplayer might behave if given an word doc file, but if it behaves badly it will be killed before too long.

* Indexing data is a strenuous job. Think about all those heavy applications which process or generate these files. But people want indexing to be as silent as possible. There are frequently recommendations that beagle should use high nice, low system priority. low IO priority etc means to be as unobtrusive as possible. The fact is, beagle already does that. However, now we even go one more step by using SCHED_BATCH scheduler policy.

There are other side improvements too, RTF filter is new. The current one is based on the legendary RTF parser by Paul Dubois. Image filters are almost new; we now have Konversation (KDE IRC client) and KOrganizer (KDE tasks and eve nts scheduler) backends. By the way, soon after 0.2.16 was released, Opera webhistory backend was added to trunk. You can just drop the binary from here into your 0.2.16 /usr/lib/beagle/Backends folder and start using it, err... trying it. I do not know how complete it is.

I would like to end by thanking the excellent user base that beagle has developed. Without them, it would not be possible to fix a whole lot of these problems. Beagle would not be what it is today without them.

Seekable LineReader

Recently I need a way in StreamReader to get the position of a line which can be stored and later skip to that line directly. C# StreamReader API does not have any way of doing this, except calling Read() repeatedly and then doing the processing yourself. Which is clumsy. Note that, StreamReader.BaseStream.Position might be wrong due to underlying bufferring. I thought this should be a fairly common requirement and indeed, many people have same question on google groups or other forums. No good answer though. One reason is, such a thing does not really make sense for arbitrary Streams, since it might not be possible to seek in them.

I needed such a thing desparately, so I created an interface:

namespace System.IO {

 // A linereader interface
 public interface LineReader {

   // Returns a position marker, which can be used to navigate the lines.
   // Some implementations may only allow moving in the forward direction.
   // Might be different from line number or file offset.
   // Should only be used for traversal.
   long Position {
     get;
     set;
   }

   // Reads and returns the next line, null if EOF
   string ReadLine ();

   // Reads the next line and returns a stringbuilder containing the line
   // The StringBuilder returned could be the same one used while reading,
   // so it should not be modified and its content might change when readline
   // is next called.
   // This is the most worst horriblest API I ever designed, for sake of speed
   // And thats why this should not be a public API.
   StringBuilder ReadLineAsStringBuilder ();

   // Skips the next line, return true if successful
   bool SkipLine ();

   // Skips required number of lines; returns actual number of lines skipped
   long SkipLines (long n);

   // Close the reader
   void Close ();
 }
}

Beagle source contains the interface and several implementations.

Fasten your seatbelts; we are ready to ship

There was a discussion going on in beagle mailing list sometime ago where I made a comment that I dont think beagle is newbie ready i.e. plug-and-play yet. Beagle fans did not understand my comment and people replied why they think I am wrong. I would be glad to be proved wrong; but all of their arguments were how they were using beagle since version x.y, how beagle was shipped and enabled by default since last z releases of some distribution and how someone is able to install beagle in a large enterprise. Duh! None of these prove that beagle is newbie ready. All they do is show that beagle works and even I know that. 0/100.

I have the feeling that some of the beagle devs and followers live in the garden of Eden surrounded by a high wall of reality. Sometimes they should go out in the streets, check the bugzilla of other distros, go through user blogs (which mostly contain complaints about how beagle does not work and how to disable it), and visit some user forums where a lot of questions are how to disable beagle from starting at startup. These are laborious jobs and not pleasing. A lot of them contain flames and invalid reasons. But almost always they are started by someone who found beagle causing trouble.

Here are some links which can make your task easy:

• Ubuntu beagle bugs
• OpenSUSE beagle bugs
• Fedora Core beagle bugs
• update Gentoo beagle bugs
  

I sometimes make the rounds and all I see are I make a point of uninstalling beagle on all my machines and The first thing I did after ... was to uninstall beagle and now my machine is happy. Silly men, how can they not like the doggy!

Subversion arrives. Finally!

KDE uses SVN for their source code management. GNOME used to use CVS till yesterday, which means beagle too was managed with CVS. I do not know the technical details. but time a again we did face technical problems with CVS. I was mostly told that life would be easier with SVN and lo behold! GNOME has switched to Subversion as its SCM (actually, still switching).

The last time this was tried by the awesome GNOME guys, they later found a glitch and had to cancel the migration. As a result I lost a commit that I made within hours of SVN migration. This time I will play safe and watch it for a few days before committing anything. If everything works out, life should be easier. Joe already cleaned up quite a bit of the unused files and directories, renamed the Evo-mail backend correctly and updated the links et al. A New Year with a clean, new repo. Sweet.

PS: There is one downside though. Joe (and others too) would like to use the SVN commit messages for creating Changelog files during creating a tarball. Which basically means others cannot observe the Changelog file between releases to figure out what was changed (neither I nor Joe updated the Changelog while committing, so this is a lame excuse). The real trouble is now I cannot write any lame jokes in my commit messages. Life would be serious now. Boo hoo.

My time with the doggie

Today I read about the Ohloh project (http://ohloh.net) and added my favourite project beagle (http://beagle-project.org) to it. I was curious what it actually does.

It took them nearly 4 hours to download and analyze the source code. But it was worth the wait. It showed some interesting statistics, like 122,885 LOC codebase, 82 direct contributors (committing in CVS) and 13 of them in last 12 months.

Just for a light comparison, Firefox has a codebase of 157,207 LOC, Amarok has 169,288 LOC and (take this) PHP 6.0 has 599,805 LOC.

It was also amusing to see my share in the project: http://ohloh.net/projects/3826/contributors/21154

beagle 0.2.14

Joe announced the release of beagle 0.1.4 few hours ago. Its a fascinating and shining new release containing exciting new features, lots of memory/speed optimizations and many bug-fixes as well. Here are the major ones which are readily visible:

• Indexes tar, gzipped-tar, bzipped-tar, gzipped and bzipped files, in the filesystem as well as in email attachments. The results show you the exact file in the archive that matched the query.
  
• Do some smart tokenizing to allow matching 001234 to a query of 1234 and better matching of file names. No more missing files.
• Beagle can find and extract data itself using its dozen or more backends. But sometimes its better for other applications to send data to beagle for indexing. Beagle had the infrastructure to act as a search/indexing service provider. The release contains an example C code to show how to do that; its pretty simple actually. Obviously python can also be used.
• Some cool signal mechanism which help to figure out what file in being currently indexed and for how long. This will be helpful if you feel beagle is taking ages to index some file.
• Use Xdg autostart mechanism to auto-start beagle. KDE4 will also implement xdg autostart mechanism. One more step towards being DE agonistic.
• The indexing information now explicitly mentions if the initial indexing is in progress. Also clients now have the option of being notified when the initial indexing ends.
• Lots of memory fixes. bhale just mentioned in the irc channel holy crap, startup RSS for beagled is 15m... beagled is below nautilus in mem usage...  im not believing my eyes :) Thank you for your myth on how beagle is a bloatware.
• API and beage-search support to know the total number of documents that matched any query. Not the superficial imposed limit of 100 documents.

Go, get it!

License to hack-debug-release

I start with the disclaimer that I am not a lawyer. I am also not very careful in reading the EULAs (End User License Agreement - a short acronym of a long term usually describing an even longer gibberish english text). When I release my first piece code (was it JBabel or mGet ?), I put them in GPL (possibly v1). Of course I didnot know what was I doing - I was merely completing a formality during the fascinating experience of sharing one's program. With time, and during development of Beagle, I came to know about the various licenses. It was about this time that the GPLv3 debate started. Also, I started releasing verious kde-beagle softwares which required me to figure out the correct licenses for them.

Liceneses, I feel, serve two purpose, demand credit where its due and specify the amount of responsibility.

Authors of some projects just want to release code without claiming any credit. I did so too. Only to be pointed out by someone that I have to actually specify the terms for others to use my code. Since I dont want to be credit for the code, its natural not to held responsible for its damages either. That needs to be specified too. One can either cook up one's own license or just pick one from the sea of licenses (http://www.opensource.org/licenses/) . The license I found to do just that is the MIT license.

MIT (X11) License: Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, ...

The license is itself not copyrighted, so it can be modified if needed. A rather free-to-do license, except for this clause following the above text

... subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

Wikipedia tells me that the latter part is necessary for the copyright laws of US and other countries. The credit part is followed by the author responsibility part

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT...

Often, people use this clause to explicitly deny responsibility

Except as contained in this notice, the name(s) of the above copyright holders shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Software without prior written authorization.

I havent seen projects where the last clause wasnt present, so I assume it is not a necessary clause. I didnot use it myself when I released yaBi under MIT license. The core parts of Beagle and Mono are released under this license.

My other project, kBeagleBar uses the fancy LGPLv2. I had to use LGPLv2 because I was using some source code from kde libraries. Note that, LGPL allowed me to use the library without any restriction, but I copied some code from a library source file - so I had to use LGPL. Similarly, I release kontrol as (distributed as a part of kbeaglebar) which uses the code from GPLed kerry, so its under GPL too. Also GPL and LGPL are copyrighted by FSF. But then again, I wouldnt dare to modify them myself.

I am looking forward to GPLv3. No, not because I have any particular reason to like it. Because I hope to read it fully, understand it and read the various articles and blogs about it to understand the issues of free software licensing. Its always good to know where my rights end and start.

Even after all this, it will take me some more years to actually read EULAs. They are frightfully convoluted to confuse my little grey cells.

Bigger than life icon

Icons are small pieces of art I encounter in my everyday life. They are like
apparels, you see them in all forms in every moment of life. They serve
different purpose, keep you warm, keep you cool, keep you comfortable
whatever but all them try to look good (some even make it their prime
agenda). In human-computer interaction, icons are large, easy targets that
serve as better visual cues than textual messages that needs to be parsed and
pattern matched. A good icon make that experience fun and enjoying.

I started with the the usual KDE crystal icon theme, and then tried different
icon sets. The tango icon theme is good, but like the usual gnome colours,
they have a very low-saturated colours which look a bit drab and boring on my
desktop. I am not looking forward to extremely gorgeous, highly saturated and
colourful icons; my desktop is not a colour palette, but the art forms I will
stare at a million times over months better be pleasant to look at. kde-looks
has quite a few icon theme, and a lot of them are mixed icon theme. For a
while, I tried KDE Crystal Diamond icons. It has good icons, except some of
the icons are very light coloured, white based and its hard to make out the
icon contents. Most of the filetype icons and the media-play (amarok)
play/pause icons are like that.

I have heard a lot about Oxygen icons; unfortunately their license does not
allow anyone to package them. Of course, I could download the svg icons
directly from their svn and convert them to an icon theme locally. I need to
install inkscape for that. Interesting enough, I found a forum post in
sabayonlinux about using oxygen icons in current kde desktop. The user gives
out the details of how to make an icon theme and later even provides a tar.gz
package to install oxygen icons. Oxygen is work in progress, so that user
replaced the missing icons by other icons. I think he violated their license
but I have to double check. Anyway, I installed that icon them and its really
good. Attention has been paid to every detail. Icons are 3d and 2d exactly
according to their need, those that look good with shadows have them, the
colours look just ok. All in all, I am pretty satisfied. I am looking forward
to manually checking out the oxygen icons and making a theme out of it. That
way, I will get the icons from oxygen as they are checked in. Good work,
guys.

Mono.FUSE Filesystem for Digikam Tags

FUSE is cool anyway. And writing FUSE filesystems in C# is even cooler. Adjectives aside, I wrote a FUSE-DigikamFS filesystem for browsing Digikam tags. Its much easier to browse the images organized by tags using konqueror or kuickshow. This way I can also share the tags mount-point by kpf (or some other HTTP server) allowing others to browse my images arranged by both folders (Albums) and tags.

A sample session,

[debajyoti@dbera Tags]$ pwd
/home/debajyoti/Tags
[debajyoti@dbera Tags]$ ls
Artistic/  Favourites/  Fireworks/  Food/
Nature/  People/  Places/  QuickCheck/  Season/
[debajyoti@dbera Tags]$ ls Artistic/
Black-White/  Mood/  Motion/  pa020099.jpg@
pa020102.jpg@  Perspective/  Shadow/

As you see, I use symlinks to overlay the images from tag filesystem. Works pretty cool.

Ed-ed

Who knew that 'vi' was based on the legendary editor 'ed'. Or, inspired by. Being a die-hard vi-fanboy, it took me no time to get used to ed, at least being able to compose/edit/save simple documents. An ed-documentation page at BUCS provided me with a quick howto listing (all) the important commands. Actually, all the commands, since there are only a few of them. Unlike vi, where you have to know thousands of commands. Its wonderful how much simpler life becomes if we focus on functionality alone without caring for presentation.
If you are comfortable with vi (or even with vim or with gvim but dont use the menu too much), I suggest you to give ed a try. You'll love it.

(From the documentation page) An ed-quickcard:

All the standard ed commands are listed below, together with a brief description of their function. Those commands which may be given with line addresses are shown with the default values of the addresses in the first column. For example, 1,$ for w means that the write command may be given with one or two line addresses to specify a particular line or a range of lines to be written to a file, and that if no address is given the default address is 1,$ (ie, all the lines in the buffer are written out). The default address . represents the current line and $ represents the last line in the buffer.

The Input Commands

Default Address  Command  Function
.                a text   Append input text after addressed
                        line.The last line input becomes the
                        current line.
.,.              c text   Replace addressed lines with input
                        text. The last line input becomes the
                        current line.
.                i text   Insert input text before the addressed
                        line. The last line input becomes the
                        current line.

The Edit Commands

Default Address   Command   Function
.,.               d         Delete the addressed lines from
                          the buffer. The line after the
                          last deleted line becomes the
                          current line.
                e file    Delete the buffer contents then
                          read file into the buffer. The
                          last line read in becomes the
                          current line.
                E file    As for e but no warning is given
                          if a modified buffer has not been
                          written out.
                f file    Print current remembered filename
                          if file not specified, otherwise
                          set it to file.
1,$               g/R/cmds  Perform cmds on all addressed lines
                          matching regular expression R. Last
                          line in which a match was found
                          becomes the current line.
.,.+1             j         Join together all the addressed
                          lines. The resulting line becomes
                          the current line.
.                 kx        Mark the addressed line with the
                          single lower case character name x.
                          The addressed line becomes the
                          current line.
.,.               l         List the addressed lines showing
                          non-printing characters and folding
                          long lines. The last line listed
                          becomes the current line.
.,.               mA        Move addressed lines to follow line
                          whose address is A. The last line
                          moved becomes the current line.
.,.               p         Print the addressed lines. The last
                          line displayed becomes current line.
                q         Exit from the editor.
                Q         Exit from the editor with no warning
                          if a modified buffer has not been
                          written out.
$                 r file    Read file into the buffer after
                          addressed line. The last line read in
                          becomes the current line.
.,.               s/R/S/{g} Substitute string S for the regular
                          expression R in the addressed lines.
                          If g specified substitution is made
                          globally throughout addressed lines.
.,.               tA        Copy the addressed lines to follow
                          line whose address is A. Last line of
                          the copy becomes the current line.
.,.               u         Undo the effect of the previous
                          substitute command. The current line is
                          reset to its value before that command.
1,$               v/R/cmds  Perform cmds on all addressed lines
                          not matching regular expression R.
1,$               w file    Write the addressed lines into named
                          file. The current line is not reset.
1,$               W file    Append the addressed lines to the
                          named file. Current line not reset.
                x         Decrypt or encrypt the text according
                          to an input key. The current line is
                          not reset.
$                 =         Print the line number of the addressed
                          line The current line is not reset.
                |cmnd     Pass cmnd to the UNIX shell to be
                          executed. Current line is not reset.
                A         Where A is one of the legal address
                          forms listed above: locate addressed
                          line and display it. The addressed line
                          becomes current line.
.+1               newline   Print the next line. Addressed line
                /linefeed becomes the current line.

Obligatory jokes link (not a PJ).