MondoSearch for Sitecore (Part 3)

As promised, I'm going to share some more screenshots of the integration between MondoSearch and Sitecore. This time I'll focus on the integration of BehaviorTracking.

BehaviorTracking Portal. The main entry to the BehaviorTracking information from within Sitecore is in the BehaviorTracking Portal, a portal somewhat similar to the well-known Sitecore Today portal, only this time the portlets filling it are BehaviorTracking portlets. Although we're still missing some of the graphics from the original BehaviorTracking this makes out a pretty decent approach to discovering what your website visitors are interested in and by double-clicking on a given keyword, it will open the BehaviorTracking Term Details for that search term.

BehaviorTracking Term Details. When you want to examine a specific search term, you can use the XAML application Term Details. Here you can look up search words, and examine

a) Which search terms are related (meaning which other terms are typically used by the same users in their searches). This can be quite helpful in inspiring new keywords for pages as well as new synonyms for the search.

b) Which pages are typically chosen from the result page, giving you a more exact idea of what the user actually meant. Use this for improving ranking of some pages, or perhaps adding a searchheader or searchname for a given page.

c) The most recent user sessions searching for this term. This might not be so useful, but it does give you that cool "Big brother" feeling :-)

Finally, you can also get BehaviorTracking Item Details. For any given item on the website that inherits from the MondoSearch Base Template, you can see a list of which search terms sent users to the various versions of this page. This is an excellent tool to optimize the content on the individual pages, to the expected content of the users.

As mentioned in Part 2 of this trilogy the along with the integration we also released some code samples, showing how to use BehaviorTracking and search to spice up your site.

On last of these examples is the "Most Wanted" list that is a small control listing the top 5 pages most often chosen from a search result page. I find this to be quite useful, as this is not the most visited pages on the website (the most visited page on a website is quite often the front page that doesn't hold any relevant information at all), but the pages that most people have been looking for. In many cases it will be quite a good help for your users to promote these pages on the front page so they can go directly to them without wasting any more time.

Improving Search with BehaviorTracking

The topic for this post is Behaviortracking (check out the website, cause I'm not gonna spend time here explaining what it is). This is a post where I'm basically gonna pretend I'm in marketing and fill You, my dear reader, with something that might resemble a sales pitch for a particular product.. "Why?" and "Where's the code?" I hear you ask. Well, first of all I feel quite strongly about this - I've seen so many websites that ought to start listening to their visitors instead of their executives - and with regards to the code...well I'm sorry no code this time.

The reason BehaviorTracking is such a cool tool, is that where other web analytics software might tell you about popularity of pages and server loads during the day, BehaviorTracking tells you exactly what you need to know: What are users looking for on my site.
In my mind it's perfectly obvious.
A website is to some degree like a shop. You have some users who browse around the shop, looking at the shelfes and eventually leaving, and some other users who go directly to the clerk at the counter and ask for a specific item. Obviously the people going directly to the counter with a specific goal are the ones most likely to buy - and naturally these are the people you want to listen to. Now, suppose you are the proud owner of a clothes store and a customer walks up to your clerk and asks for a specific pair of "Levis" jeans. Would you like that answer to be:
a) duuh
b) I'm sorry, I don't know anything
c) The jeans department is over there
d) Here is a number of Levis jeans that should fit you, this pair is very popular and this pair here is on sale this week. By the way could I also interest you with a new shirt that matches to go with that?
e) We don't have any Levis jeans at the moment, but I'll make sure to order some. Meanwhile perhaps you'd like to check out this competing brand that looks similar and is a bit cheaper?

(I'm no sales guy, but I could imagine two of the above answers being good - you figure out which).
A good search engine is like a good sales guy greeting people at your store, helping them while selling your products. But in order to always provide the best assistance it needs constant optimization - and thats where BehaviorTracking comes into play. By frequently examining the search patterns of the visitors it's easy to customize not only the website but also the search engine to provide the best possible service to your visitors.

MondoSearch for Sitecore (part 2)

As earlier promised, here's some more info on v. 1.1 of the integration between MondoSearch product suite and the Sitecore CMS system, that was released just before christmas. In this second part of my story I will focus on the search itself and the ways it has been integrated.

The point of the integration was to integrate not only the search engine but also search analytics, crawler administration into Sitecore, making Sitecore a common user interface for both products.

The reason is simple. Although website search over the last couple of years has become increasingly commoditized it's not just something you plug in once, and then expect to have working perfectly ever after. Search is a dynamic thing - like the website it indexes and for the best end-user experience it should be continiusly tweaked and improved to match the expectations of the end-users. The ideal way to do this is by studying the behavior of the users and then optimize both website and search for them (I could talk for hours about this subject, but I'll safe that for another post). Nevertheless that makes it even more important to make the Search and Behavior analytics easy to use for the webmaster/marketing dept. responsible for a given website - and hence we decided to go for as complete an integration between the products as possible.

The search part of the integration includes:

3 Search Result Sublayouts, all based on a Search Template. All of the support Sitecore authorization enabling them to only show the results the logged-in user is allowed to see. All the texts used on the templates is defined in the template, so it's easy to translate in Sitecore. The Sublayouts use the standard MondoSearch SearchTemplate technology so it's easy to change look & feel and add functionality.














2 SearchBox sublayouts, simple and advanced that can be placed on any layout to enable the possibility to search.

Click Item and corresponding layout, enabling logging and highlighting of search results.


A Meta-data xslt rendering for sending item-related meta-data to MondoSearch.



A Base template that allows Sitecore items to have fields to hold meta-data for MondoSearch, including Search categories and indexing rules.














A Crawler Control XAML application that allows an administrator start and stop the MondoSearch crawler as well as publising crawled databases. This tool will also display the current status of the crawler, crawler log and number of indexed pages.


A Sitecore task for starting the crawler so the Sitecore scheduler can be used to scheduling crawls.


An Editor Search XAML application that allows Sitecore editors access to use MondoSearch to find the items they want to edit. When a result is selected it will of course open in the Content Editor for easy editing.



Templates and items for defining Categories used in Search.



MondoSearch Examples
On top of the integration Mondosoft also supply some coding examples of how to improve the overall functionality on the website. Like this Autocomplete search box that uses frequently searched words as autocomplete suggestions that appear while you type a search query.




One of the other examples is a "Related Pages" box that will use the search engine to search for other related pages to the current page, and "Related Topics" that will use Behavior Tracking to suggest search terms relevant for the page you are currently on.

Now, this was just a brief overview of the "search part" of the integration. In the next post I'll go through all the new cool features the integration adds to Sitecore to track visitor behavior and search term popularity.
Later on I'll also show how the it's possible to add SearchHeaders (custom html/sponsored links) to the search results from within Sitecore and outline a couple of ideas I have on how to further improve the overall value of a Sitecore website.

The new update of the Integration demo-site is due to be launched any day now and it'll be possible for all interested to try out these features on their own - either on the demo-site, or by downloading the integration.

AI Efficient Programming

Today I started on a new course on ITU, entitled AI Artificial Programming. I've often before played a little around with various ML mechanisms, Neural Networks, m.m. but never studied AI in a structured mannor.. So far the course looks very promising.
I bought a couple of books for the course that both seem really good:

• Artificial Intelligence: A Modern Approach (2nd Edition), Russel & Norvig
  
• AI Application Programming, Jones

As soon as I find my feet in this course I'll post some of the silly AI thingies I'm destined to make :-)

Blogspot hint: Google optimization

I just implemented this little optimization of the blog in the hope that it'll improve Google's search results. The problem I noticed that if you find this blog through a search on google, you'll quite often get a link to the front page (allantech.blogspot.com/index.html) in your search results because a related article was on the front page at the time when google indexed the site. However with the current update rate it's also quite likely that when you click that link, the article is removed from the front page already. An example: today I searched for "AllanZip" and this is the result I got:Notice how you are directed to the front page instead of the article page.
The way to avoid this is to put a robots meta-tag on the front page, instructing google to "noindex,follow" meaning "don't index this page but follow the links" - however this shouldn't be put on the item pages. Since BlogSpot uses the same template for both the main page and the item pages, this took a little bit of research, but finally I got this code to do the trick:

<b:if cond='data:blog.pageType != "item"'>
  <meta content='noindex,follow' name='robots'/>
</b:if>

I hope this little trick will improve the search results - let's see when Googlebot will honor me with yet another visit :-)

Zip Online

I was just briefly playing around (again) with ICsharpCode's SharpZipLib and I made this little tool that I figured might come in handy some day.

It's an online zip opener. It's handy in the cases when you are asked to download a zip file, but you really don't feel like filling up your desktop with yet another zip when all you need is a single file for the content. Or perhaps for the cases where you'd like to link directly to a file within a zip. Or for when you want to index a zip file (as always I think in search) :-)

It's basically a web app, that takes two parameters: "zip" with a url to the zip file you want to examine and "file" for the file name/path within that zipfile that you want to extract.
In case you only give the "zip" parameter it will list the content of the zip file.

The most important method is this (and yes, I realize the code isn't pretty - it's just a simple poc):

       protected void LoadZip(string url, string file)
       {
           Error.Text = "";
           HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
           HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
           if (resp.ContentType == "application/x-zip-compressed")
           {
               if (resp.ContentLength < 2000000)
               {
                   BinaryReader br = new BinaryReader(resp.GetResponseStream());
                   byte[] buf = br.ReadBytes((int)resp.ContentLength);
                   br.Close();

                   System.IO.MemoryStream ms = new System.IO.MemoryStream(buf);
                   ZipFile zf = new ICSharpCode.SharpZipLib.Zip.ZipFile(ms);

                   if (file != null)
                   {
                       ZipEntry ze = zf.GetEntry(file);
                       Stream filestream = zf.GetInputStream(ze);
                       BinaryReader br2 = new BinaryReader(filestream);
                       byte[] newbuf = br2.ReadBytes((int)ze.Size);
                       br2.Close();
                       Response.ClearContent();
                       Response.ContentType = MimeType(file);
                       BinaryWriter bw2 = new BinaryWriter(Response.OutputStream);
                       bw2.Write(newbuf);
                       bw2.Close();
                       Response.End();
                      
                   }
                   else
                   {
                       GridView1.DataSource = zf;
                       GridView1.DataBind();
                   }
                   zf.Close();
               }
               else
               {
                   Error.Text = "Zip file too big";
                   resp.Close();
               }
           }
           else
           {
               Error.Text = "Wrong content type: " + resp.ContentType;
               resp.Close();
           }
       }

A couple of parts was a bit tricky. First the part that loads the external zip into a memorystream for easy access by the sharpziplib. It turns out that the BinaryReader was quite necessary in order to load it without corrupting the data. Also determining which mimetype a given document in the zip file is, before sending it out wasn't all that easy, but in the end I found a solution online. This solution looks up mimetypes in the registry so it does require registry access for the aspnet user - that can potentially be a problem.
There's several known problems with this implementation, but I'll leave it to you guys to figure them out :-)

Download the code here.
Try it here (note that my "playground" server where this is on might be closed down pretty soon - enjoy while you can).

What to watch.......KISS EPG

For the last year and a half I've been the proud owner of the ultra-cool (nevermind the bugs) KISS DP-558 harddisk recorder.
The really cool thing about the dp-558 is it's ability to go online and retrieve program guides, so you easily can see what's on the telly, and ask it to record it. It's even possible to schedule it to always record your favourite show every time it's on. If you download 3rd party software for the kiss it's also possible to use it as a webserver/ftp-server and much much more. All in all a cool product in spite of it's flaws.
But after KISS technology got aquired by Linksys it has really been going downhill. Recently they changed the provider of the Electronic Program Guide (EPG) to a service called TVTV. This new implementation is so filled with bugs and problems that I'm surprised it passed through their QA dept.

Check out this part of today's program guide for instance. I do admit that typically the DR2 channes has a lot of reruns....but comon 5 reruns of the same movie at the same time?! Well..not to worry. If I get frustrated watching all 5 movies at the same time, I can always switch to KAN4 or TV3 that apperantly doesn't broadcast today after all...Oh how I look forward to the tranquility of a black screen :-)

Clever minds think alike

Just as I was checking the google rank on my posts about the upcoming Poker challenge, I found this article on codeproject. Isn't it typical that whenever you get a great idea somebody else has already gotten the exact same idea and implemented it?
In fact even the implementation of code is somewhat similar to what I've already coded.
Well - this just goes to show that clever minds do think alike :-)

And of course this doesn't make me change my plans - I'll keep going with this little poker challenge - I'm sure I can put a new twist to it!

I HATE DEALIO

You always think bad things never happen to you, only to the ones next door. Naturally I got a shock today when I opened my IE7 and found that an additional ad-filled toolbar had installed itself.. This time it was a toolbar called Dealio that had gotten snuck past my F-Secure and infested my system with it's ads..."No Spyware" it claimed..yeah right - I always believe what uninvited virus-like programs tells me...
When will makes of malicious software like Dealio (and companies advertising through them) learn that it's just not cool?! I suppose it could be a good idea to make an online blacklist of companies buying adspace in malicious software - perhaps that would teach them a lesson...But then again, a part of me just feels like the best solution might be to leave the culprits to the justice system - and hope the judge has had his PC infected recently....Nothing says "please stop coding adware" like 20-30 years of hard labor :-)
Surprisingly Dealio seemed easy to uninstall....almost too easy..I wonder what I've gotten myself into - it certainly doesn't seem like F-Secure will be any help in keeping the evil-doers out. After I uninstalled I was sent to www.dealio.com where I got to fill out a survey of how well I liked Dealio - these guys really have some nerve!

On-the-fly conversion of MSOffice to raw text

So, this is what I needed the HTTPModule for: a HTTPModule that on the fly converts office (and other weird formats) to text/html. I can imagine several scenarios where this can come in useful (but I'll keep all the great ideas to myself for now).
The POC is something like this: a user requests a non-HTML document (office, pdf, etc) located on a website with "convert=true" in the querystring (http://myserver/test.doc?convert=true). Then the document is automatically converted to text and returned.

In order to do this we'll use ifilters, and to easily access them from C# I found this neat little library by Eyal Post on codeproject.

The code for my module is here:

using System;
using System.Web;
using EPocalipse.IFilter;
using System.IO;

namespace Allan.Tools
{
  public class IFilterModule : IHttpModule
  {

      public void Init(System.Web.HttpApplication application)
      {
          application.AuthorizeRequest +=
             new EventHandler(this.Application_AuthorizeRequest);
      }


      private void Application_AuthorizeRequest(object sender, System.EventArgs e)
      {
          HttpApplication app = ((HttpApplication)(sender));
          HttpContext context = app.Context;

          if (context.Request.QueryString["convert"] != null)
          {
              TextReader tr = new FilterReader(context.Request.PhysicalPath);
              string s = tr.ReadToEnd();
              tr.Close();
              context.Response.Write(s);
              context.Response.End();
          }
      }


      public void Dispose()
      {
      }
  }
}

In order to build it, download Eyal's library and reference it in the project. Then after building the module include it in your web.config like this:

<httpModules>
<add type="Allan.Tools.IFilterModule,IFilterHTTPModuleTest" name="IFilterModule"/>
</httpModules>

Enjoy!

Cool tool: Generate HTTPHandlers and HTTPModules

I was just looking into custom HTTPHandlers and HTTPModules and I came across this really cool tool made by Milan Negovan. Although it looks like the tools is kinda old it just gave me a good starting point for making a custom HTTPModule (I'll probably post what I needed it for in a later post).

Update: The Poker Challenge

I now have some more exciting details to share on the upcoming Poker challenge (although dates are still unknown).

I decided to change the strategy a bit and instead of a web-service solution (sorry, microsoft buzz word dept.) go with a more object oriented approach.
Those of you who participated in Microsoft's .NET Terrarium might recognize the path I'm planning...
I will on this blog release a poker library, which includes all the classes needed for a nice game of poker (Texas Hold'em). It'll have Card class, Game class (for keeping track of players and betting), CardCollection (that can calculate what kind of a hand you have), etc. It will even have a Player class - but this is where you come in! That class will be abstract. So, the challenge is to use make your own Player class, inheriting from the base Player class, then code the behavior that makes your poker player the best in the bunch. While developing your player you can try it using a small test-stub I'll include, and when done you simply upload the dll to a website (details follow later). Then, we'll hold match(es), monitor them online real-time (hopefully) and find a winner!
Still havn't heard from anybody willing to sponsor any winnings - but there is still time.

MondoSearch for Sitecore (part 1)

As my previous post today might indicate I've been spending my time setting up a public demo-server for the MondoSearch-Sitecore integration.

Even though I was project-manager on the integration project, I must admit that I'm a once again a bit surprised (read: proud) how smooth and easy the integration works once both Sitecore and MondoSearch is installed and working.

It just takes a few steps like installing a package in Sitecore, adjusting web.config, and doing a little customization in your Sitecore website for neatness.

It's a pretty full integration, integration both Search, Crawler Administration, Search BehaviorTracking and Marketing tools for optimizing search within the Sitecore admin UI.

I just need to tweak a few more details to make the website really fit for public display - and as soon as that's done I'll post a lot of screenshots here about it.

For now you will have to settle with this screenshot of the MondoSearch Crawler Control as a Sitecore XAML application.

Permissions. Can't live with them, can't remove them

Here's an annoying little problem that has been pestering my life for a good day or so. On a newly installed win2k3 server with SQL Express 2005 I was trying to install the Sitecore Demo-site "Printers Inc.". Since the demo-site uses attached databases to SQL Express 2005 it was supposedly fairly easy to install: unpack a zip file, setup IIS, set the right login/password for your connectionstring in a config file.
All done, and try to open the site, but get greeted with this message:

Server Error in '/' Application.

---

Unable to open the physical file "D:\sites\Sitecore\MySite\Databases\sc53Master_Data.mdf". Operating system error 5: "5(Access is denied.)".
An attempt to attach an auto-named database for file D:\sites\Sitecore\MySite\Databases\sc53Master_Data.mdf failed. A database with the same name exists, or specified file cannot be opened, or it is located on UNC share.

Okay...Looks like a permissions thing I thought...So I hurry and give ASPNET and NETWORK SERVICE users full rights on the entire site, and set it to propagate rights to children (security, who needs it anyway?!). Still the same problem....At this point I actually start reading the error message more carefully..Oh, it actually suggest 3 possible error scenarios...well - let me see: It's on a local disk, so forget about the UNC thing. A visit to SQL Server management studio proofs that there's no database with the same name (why would there be? it's a brand new server). And I had just made sure that the permissions were right.
At this point I naturally went to the no.1 problem-fix with microsoft products - but even a full reboot didn't seem to do the trick...Now what?! Checked with the local SQL-server wizards and our sysadmin guru....posted the problem on SDN5 Forum...still no luck.
After wasting too much time I (we) reached the point of desperation where you try out stuff that just doesn't make sense in trying to solve this problem - like setting up the SQL Express service to run as "Local System" instead of "Network Service". And what do you know - it paid off...all of a sudden everything worked splendidly. Still doesn't make sense to me - Network Service had full permissions for the databases....arggh.

AllanZip revisited

I can't even begin to count the times I've sat in a dark room in front of an old monochrome monitor (in my early childhood) enchanted by the magic of such amazing applications as pkzip, arc, lzh, rar, and so on. Imagine taking a certain amount of data, compressing it to almost nothing without loosing any information and easily being able to extract it again. After some time, the internet came along and I started reading up on lossless compression in various newsgroups. I gazzled at stories of lossless compression programs that could compress any data to 90% of it's former size, so you'd just have to run the program enough times to reduce any amount of data to almost nothing.....Naturally I quickly figured out that it was just hoax - but I was still left trying to figure out how the actual algorithms for lossless compression were constructed.

Eventually I became curious if I would be able to make my own compression program so I decided to give it a try. After checking the books I decided that I could try to implement the Huffman algorithm for fun and see where that got me. Today I came across that old patch of code, and ported it to c# 2.0 for your viewing pleasure :-)

If you, my beloved reader, is a data-dummy, here's a quick intro to the concept of lossless compression:
Most uncompressed data is stored pretty simple. The data consist of bytes, and each byte consist of 8 bit (0's and 1's). A character is often a byte (although some encodings use two bytes per character) making it possible to have 2^8 = 256 different characters. However this way of storing data isn't all that optimal since not all characters occur as frequently as the others. The concept of most compression algorithms is to find out what characters are the most popular and describe those in the fewest bits possible, while then using more than 8 bits to store the most uncommon characters.

My small code demo works in several steps. First of all it breaks up the text to compress into characters, creating an object for each (in this case called a "LeafNode"). Then it counts how popular each character is and assigns it as a weight.

     private void BuildTree()
     {
         //Prep leaves for tree
         LeafNode[] leafs = new LeafNode[rawList.Count];
         this.rawList.Values.CopyTo(leafs, 0);
         Array.Sort<LeafNode>(leafs);

         //Make Queue's
         Queue<LeafNode> q1 = new Queue<LeafNode>(leafs.Length);
         Queue<InternalNode> q2 = new Queue<InternalNode>(leafs.Length - 1);
         foreach (LeafNode ln in leafs) q1.Enqueue(ln);

         //Build tree
         while ((q1.Count > 0) || (q2.Count > 1))
         {
             HuffNode hn1;
             HuffNode hn2;
             //Get the two smallest nodes from either queue.
             if ((q1.Count > 0) && (q2.Count > 0))
                 hn1 = (q2.Peek() < q1.Peek()) ?
                     (HuffNode)q2.Dequeue() : (HuffNode)q1.Dequeue();
             else if (q1.Count > 0) hn1 = q1.Dequeue();
             else hn1 = q2.Dequeue();
             if ((q1.Count > 0) && (q2.Count > 0))
                 hn2 = (q2.Peek() < q1.Peek()) ?
                     (HuffNode)q2.Dequeue() : (HuffNode)q1.Dequeue();
             else if (q1.Count > 0) hn2 = q1.Dequeue();
             else hn2 = q2.Dequeue();

             //Create a new internalnode pointing to the two found and enqueue it.
             q2.Enqueue(new InternalNode(hn1, hn2));
         }

         TreeRoot = q2.Dequeue();
     }

Afterwards it builds up a binary tree of the characters, based on popularity (the most popular has the shortest distance from the top). From this tree you can extract the bits needed for each character when compressed.

The binary tree showed here is for the text: "Lossless compression is lots of fun". Notice for instance how the "s" character that occurs 8 times has a very short binary signature 00 while "r" that only occurs once has the signature 11101.


When this tree is ready, it's just to look up each LeafNode and find it's bitstring (by recursivly examining the parents) and append the bits to your compressed output. You are effectively looking down-up. When you want to decompress a compressed string, you just need to do the opposite - start from the top, follow the bit-patterns down until you reach a character. Then start from the top with the next bit again.
The encoding code looks like this:

          //Build conversiontable
          this.rawList.Clear();
          //Make a frequency list of chars
          this.InsertChars(s.ToCharArray());

          BuildTree();

          //Temp: output tree
          OutputTreeAsHTML();
        
          //Encode string
          StringBuilder sb = new StringBuilder();
          foreach (char c in s)
          {
              sb.Append(rawList[c].GetBits());
          }
          string myBitString = sb.ToString();

When I execute the included console program, that first compresses and then decompresses the above string I get the following output:

As you can see we managed to compress to 43% of original size (35 bytes went down to 15). However note that in this compression we did not include a description of the tree, which would be necessary to decompress it.

I remember that I played around with this for a while back in the day and I even wrote a small commandline util that would compress and decompress a file. However after I did some compressions tests, comparing my implementation with Zip, it all of a sudden wasn't that fun any more :-)

Download the full C# 2.0 source code sample here.

"Manually" importing Sitecore Items

Well back from Ukraine, I better deserve the kind mention on Alexey's blog with a couple of Sitecore hints.

As many others I've on occassions spend some time struggling with packages in Sitecore. Especially with large packages, the scaling of the build-in packager doesn't seem optimal (at least in versions prior to 5.2.0.12), and when running through a webbased environment the entire installation process of a large package could easily fail do to browser / IIS timeouts.
All in all, I must admit to having been a bit annoyed with the entire webbased architecture of the packager - it was difficult to generate a package as part of an automated build-process, it was difficult to work with programmatically and as mentioned, it would crash often (it does seem to look somewhat better in Sitecore 5.3). So naturally I set out on a small quest to make my own console-based packager for Sitecore, however still based on the same package format (zip files with a mixture of files and xml-items). The goal of this tool was to run in a build-test enviroment for packages doing the following:

1. After a new compilations of the components of a package is completed, extract the items and files necessary and make a package automatically
2. Automatically install a number of packages on a dev-server to prepare it for regression-testing.
3. A tool for manually syncronising certain items in Sitecore

I examined the architecture of the packages a bit and decided that it would probably be easiest to let my source-control system handle the files, extract them to a "files" folder, then add the items extracted through the Sitecore WebService (from a list of items defined in a source-controlled file) and put them as xml files in an "items" folder, then add meta-data and zip the two folders. However I never got my solution working - my best guess is that zip-format used by Sitecore isn't standard. This could probably be brought to work - but not in the time-frame I had available.

However for the second task I did manage to create a small, but ugly, tool that could assist in manually installing packages (the ordinary packager crashing was a good motivation here).
Since I never completed the tool, I won't put it here for download, but let me just share the code-bit that uses the Sitecore webservice (located at "/sitecore/shell/webservice/service.asmx") for updating items. Perhaps it will be of use to someone out there facing the same problems as I was.
To manually install items extract the items from a sitecore (5.2) package using a zip-program. Then run this code on the folder containing the items.

                   string importpath = args[1];
                   ws.Credentials c = new SitecoreItemExport.ws.Credentials();
                   c.UserName = Properties.Settings.Default.Username;
                   c.Password = Properties.Settings.Default.Password;
                   ws.VisualSitecoreService service = new SitecoreItemExport.ws.VisualSitecoreService();
                   List<String> Paths = new List<string>();
                   GetPaths(importpath, ref Paths);
                   foreach (string p in Paths.ToArray())
                   {
                       StreamReader sr=File.OpenText(p);
                       string xml = sr.ReadToEnd();
                       sr.Close();
                       string db = "";
                       string id = "";
                       ExamineItem( p, ref db, ref id);
                       XmlNode xn=service.InsertXML(id, xml, false, db, c);
                       if (xn.InnerText == "ok") Console.WriteLine("Success: " + p);
                       else Console.WriteLine(xn.InnerText+" "+p);
                   }

The "ws" is a namespace pointing to a web service reference, the credentials used is the admin login for Sitecore and it uses the method "ExamineItem" for some simple string manipulation (getting the database and db-item-path from the file-path of the item xml file).

And I'm off to Ukraine once again

Next week I'm in Kyiv (Kiev) monday-thursday. My company has a small but effective (!) development office there, and I'm going to discuss a new exiting project with the guys, and hopefully also have a lot of nice Ukrainian food :-)

I'm not sure if I'll have time to post anything while in Ukraine, but you, dear reader (or Googlebot) feel free to spend the waiting time browsing some of the pics from last time I was in Ukraine.

Gentlemen, Let's Play Poker!

It has been a dream of mine for a long time to make or participate in a little coding contest - but I havn't really been able to think of a good topic for it - until now. A couple of weeks ago we were a small group of friends gathered for one of our usual Texas Hold'em poker nights.
As usual I went home empty handed - but this time I at least had an idea in my head....The ideal topic for a coding contest is something as simple as a poker game. But this time, the game should be played between coded bots, and we'll replace the casino with a webservice.

I've begun making the webservice and a good basic library for Pokerbots - now all I need are developers, a grand prize and a couple other minor details.

Anyone interested in participating in this, drop a comment to this post, please. I'll be back with more info later on rules, time, etc. Let the best Bot win!

I want these toys...

I must admit that I love gadgets - just because they are gadgets.....If it wasn't because my wife probably will complain I'd go and buy these gadget instantly :-)

The USB Missile launcher at the Gadgets.dk.
Hey a guy needs to defend himself at the office... And imagine the possibilities if this could be accessed through a coding interface.... The build-server encounters an error during the unit-testing (without which no build is ever really complete), quickly identifies the method failing, using custom attributes on that method identifies the developer responsible, looks up his desk-coordinates in the office, connects to a vo-ip service and calls his phone (to get him to his desk) and finally launches a full batch of missiles towards him :-) Wouldn't it just be wonderfully geeky way to avoid all those annoying bugs??? (can anybody notice that I've started getting less coding and more project management task in my current job? The hair I have left is starting to get a little pointy)

Another treat at the Gadgets store is this Dr.Evil USB hub. Isn't it cool? In order for it to operate you'll need to turn the key, flip the switches and if you open the glass and press the button it'll make an incredible sound (perhaps this should in fact be used in conjunction with the missile launcher?)

The only thing that bothers me is that it's merely a USB-hub....After I've shaved myself bald, gotten a monocle and a white cat and I feel like doing a bit of world domination, the maximum thing I'll achieve from this is to turn on a USB powered Fan. Again - I wish for a coding interface....Imagine connecting this thing to the build-server! Forget about continuous Integration - You'll have everybody sitting on pins just waiting to make a new build because it'll feel so cool :-)

So if anybody enjoys reading this post and feels like getting me a present, here's some good ideas.

Adding web applications to a Sitecore IIS site

A problem one often faces when trying to install 3rd party web software (like MondoSearch) on a Sitecore website, is that any non-Sitecore related virtual directory or web-application place on the same IIS website will run into a lot of problems.
This is naturally due to the httpModules, httpHandlers and roleManagers set by the Sitecore Web.Config in the root of the website.
The easiest way to get around this (besides installing on another IIS Site, which can be difficult on an XP developer machine) is to put a web.config in the non-Sitecore related web-application that removes all of the modules, managers and handlers set by Sitecore.

For a default installation of the Sitecore 5.3 demo-site, Printers Inc. this would be:

   <httpModules>
     <remove name="SitecoreHttpModule" />
     <remove name="SitecoreUploadWatcher" />
     <remove name="SitecoreXslWatcher" />
     <remove name="SitecoreLayoutWatcher" />
     <remove name="SitecoreConfigWatcher" />
     <remove name="StatCenterPersonalizer" />
   </httpModules>
   <roleManager enabled="false">
         <providers>
           <clear />
         </providers>
   </roleManager>
   <httpHandlers>
     <remove verb="*" path="sitecore_media.ashx"/>
   </httpHandlers>    

In other installations / versions this might vary a bit, but you can always check the sitecore web.config and see whats added, so you can then remove it.

"My first card game": Cassino

No, it's not my spellchecker that has suffered a nervous breakdown (although I wouldn't blame it if it did), I actually mean Cassino. You might not be familiar with this game, but it's quite a fun little math based card game, and since it's the first "real" card game I remember learning, I choose it some years ago to be the first card game I'd try to code. Coding for fun you know - not the "I-wanna-take-over-the- entire-gaming-industry" kind of code. So code I did.

For some years you've been able to see the result on cassino.thraen.dk (feel free to play). The game is pretty simple: 2 players get 4 cards each, plus 4 cards goes face up on the table. The players then take turn trying to gather points by matching table cards with cards on their hands, based on values. Read the detailed rules on Wikipedia.

I did the development in a couple of steps:

First I coded a general cassino library which contained classes for all the typical card game objects - like card, deck, hand, card-collection, etc. Then I also added some classes more specialized for this specific game like a cassino game class, card combination class, a computer player class and a point collection class (Note: this was back in the days of .NET framework 1.0, so I had to make custom collection classes to keep the code tidy. Imagine how it was to live in a world without Generics - incredible how we survived). I made the cards IComparable so it would be easy to sort a hand and check legal/illegal moves.

       //Deal
       for(int i=0;i<4;i++)
       {
           Player.AddCard(deck.DrawCard());
           Computer.AddCard(deck.DrawCard());
       }

The "computer intelligence" of the game is a simple algorithm that identifies all the options available to the computer and then makes a partially random decission which option to choose.
Afterwards I coded a webUI to it, pretty basic and pretty ugly. The only partially interesting thing there is the aspx page I made to show when cards on the table were combined. I needed something that would create the graphic of cards being located on top of each other, so I needed to dynamically merge the card graphics together. It resulted in this Page Load code for the CombiImage.aspx file:

    private void Page_Load(object sender, System.EventArgs e)
    {
        Response.ContentType="image/gif";

        //Request format= cards=xx.gif-yy.gif-zz.gif
        string cardlist=(string)Request["cards"];
        //cardlist="2c.gif-2d.gif-ac.gif";
        string[] cardsrc=cardlist.Split(new char[]{'-'});

        //Load cards
        Bitmap[] cards=new Bitmap[cardsrc.Length];
        for(int i=0;i<cards.Length;i++) cards[i]=(Bitmap) Bitmap.FromFile(Server.MapPath("Cards/"+cardsrc[i]));

        int max=30;
        Bitmap b=new Bitmap(cards[0].Width,cards[0].Height+((cards.Length-1)*max));

        //Try a raw copy
        int y=0;
        for(int cur=0;cur<cards.Length;cur++)
            for(int i=0;i<((cur!=(cards.Length-1))?max:cards[cur].Height);i++)
            {
                for(int x=0;x<cards[cur].Width;x++)
                    b.SetPixel(x,y,cards[cur].GetPixel(x,i));
                y++;
            }
    
        Bitmap b2=new Bitmap((System.Drawing.Image)b,cards[0].Width,cards[0].Height);

        b2.Save(Response.OutputStream,System.Drawing.Imaging.ImageFormat.Gif);
        Response.End();
    }

It's always fun to sit a several years later and look through some old code you've written...I always go through several phases.....pride, embarrasment, frustration and eventually I get so inspired by the ideas that I end up rewriting it all using everything I've learned since then and all the technology that has appeared in the mean time. Perhaps this little game will suffer the same fate - now when I look at it I get an itch in my fingers to clean up the code, add some better computer intelligence, AJAX'ify the UI and perhaps even add features like TopScore list, player-player games, etc. Drop a comment if you think it's a good idea :-)

And oh yeah - if you want to look at the old library I made (for a good laugh) you can download it here.

The RSA Challenge

I was just browsing through some more of my old projects and came upon this crazy little thing. If I recall correctly, some years ago I was surfing the net as usual and I began to read about the RSA challenge. I've always been quite fascinated with mathematics, numbers and especially prime numbers and perfect numbers....In fact some of my earliest childhood memories are of prime-related programming projects and later on spend countless cpu-hours on the mersenne project, looking for new big primes...but I digress.
Anyway, back when I started reading about the RSA challenge and the stories about the numbers already factored I quickly realized that I'd have probably as much chance of winning one of the prices there as a blind man would have of finding a needle in a haystack. But even so, he most definetly won't find it if he isn't even looking. So naturally I began my search...


The $20.000 RSA challenge award was paid november 2005 to a research team that had spend around 30 2.2 GHz Opteron CPU years finding the prime factors to a 640-bit number. Since I don't have that kind of time or CPU power on my hands (nor the algorithms / mathematical knowledge) I decided on another approach: I was going to play in the big Prime Lottery.
I made a small windows application that would pick out random numbers (pseudo primes) around the right size as I could imagine a factor to a challenge number would be, and then check if they were in fact a factor. Crazy? perhaps a bit when I think about it retrospective. Hopeless? Yes indeed.
I've always believed that people who buy a lottery ticket for "lotto", danish state lottery are in fact just paying an extra tax for being bad at maths....Yet the probability for them of winning the big price is probably bigger than mine at finding a prime. Anway it was fun coding and I hope you'll get a laugh out of playing with it.

The application uses the BigInteger class from Chew Keong TAN, which can be found on http://www.codeproject.com/csharp/biginteger.asp.

Download my RSALottery app here. The zip contains both the VS solution as well as a compiled executable.

Thank You Linden Lab!

What a wonderful present to get in this otherwise moist and boring january! The company behind SecondLife (one of my favorite passtimes) has open-sourced the code for the SecondLife client. This is a really important step in my opinion.

SecondLife has already for some time been one of the most advanced and cool virtual worlds, but for some time I had been fearing it's doom due to the proprietary software and protocol it used.
Now, when it's been open-sourced I can't wait to see what will come from all my fellow geeks.
One obvious project I have been considering for some time is to try to harvest some of the information within SecondLife and try to index / cluster it with a decent search tool (who knows, perhaps I'll even give MondoSearch a try here) - cause the search functionality within second-life has always left me wanting more (although the current one is relatively usable, it's merely an advanced database lookup, nothing like a free-text search engine which could be ideal to navigate this entire world of structured and unstructed information). So far I havn't been able to identify if the code-parts that are now open-source will allow me to access all the information - but as soon as I have time I'll definetly dig into that. This could very well be a new business area for my little bizz, "Pear Computers".

I'm convinced this is a major step in SecondLife's attempt to take over mine (and others) first lifes :-)

Programmatically accessing the stackt race

Every once in a while I find myself exploring new parts of otherwise well-known namespaces. Some time ago I came across the System.Diagnostics.StackTrace. This is an awesome class that gives you full reflection access to the current stacktrace.
In real life this means that you can programmatically access all the methods that are currently executing leading you to where you are now in your code. Like this, for instance:

System.Diagnostics.StackTrace st=new System.Diagnostics.StackTrace();
Console.WriteLine(st.FrameCount.ToString());
for(int i=0;i<st.framecount;i++)
  Console.WriteLine(st.GetFrame(i).GetMethod().Name);

Now, when I first stumbled across this, I instantly had a gut-feeling that this would come in handy some time. First thing that naturally pops into mind is error-handling and logging - but then it occurred to me "hey, you already get the stack-trace when there's an exception".

Somehow I still can't let it totally go - it must be useful for something else - perhaps in an extremely modularized system - or maybe for some advanced unit-testing. Anyone has any ideas?

Is AJAX the spice that makes search taste better?

Last spring I wrote a 7,5 ECTS project at ITU along with a friend, Peter Madsen, on AJAX and it's usefullness for improving functionality in a web application - in this case website search.
It was kinda inspired by all the hype around Web 2.0 and the cool features AJAX technologies allows you to do on a webpage.
It's really fascinating the way the web is moving from pages with information towards applications with communication.
To try out the technology we setup two identical search & result pages, that searches on a mondosearch which have indexed www.itu.dk. One of them we did our best to AJAX enable (narrowing search results on the fly, no full post-backs, scrollbar navigation).
Check out the samples here. If you have difficulty sleeping and feel like reading the end report, download it from here.

Later on I've been quite interested in all the possibilities there is in using AJAX along with data from user behavior to improve search - one example could be a feature like an autocomplete drop-down in the search field that suggests commonly searched queries (yes I do realize that this feature has gotten quite popular several places after Google introduced Google Suggest).

The end conclusion of our report? We kind of agree that AJAX is definetly a cool technology that can help extend the functionalities on current old-tech html pages....But the downside is that it's ugly as hell - using tons of different technologies to interact in a spider-web-chaos. Whats really needed is a new web-architecture thats intended for this use and thoroughly designed - not a big rag of patchwork upon patchwork.
This being said, I still won't hesitate to use AJAX to spice up my web-applications in the future.

Back to blogging

I have through several years thought about how cool it could be to start doing a technology blog since I spend most of my time exploring information technology anyway, and I've often been lurking and learning from other cool technology blogs like Alexeys Blog, Alex de Groot's blog, Mike Tech, Jespers Blog, Lars Nielsen's Blog, and many many more.

Eventually I started "my first blog" on www.thraen.dk. Somehow it quickly turned into more of a personal announcements/ travel pictures place than a real technology blog - I guess the two worlds can be hard to combine.

Well anyway - here I go again....I truly hope that I'll find time to keep this blog running with lots of interesting tech stuff.....If you see anything thats usefull to you, please leave a comment so I'll know whether to continue writing this blog or give up.

Perhaps I should briefly introduce myself - geek-wise: I'm a senior software engineer with a company called Mondosoft, an ISV that focusses it's products around website search, intranet search, search behavior and information access in general. I've been there for 6 years and my interests within the company span pretty far. From designing new products, researching, coding UI's as well as stuff deeper in the core, project management and occassionally I get to travel around training partners in how they should deploy and use our excellent website search engine.
I'm really passionate in most aspects around information retrieval and good usability when it comes to search and I love to spend hours trying out ideas to new functionality or ways of improving usability.

The first couple of posts I plan to have follow this will be a couple of older projects I've done that I feel like showcasing.

Cheers,

Allan