A simple, little web load tool

There are many ways of doing performance testing of web applications. In the good ol’ days I remember starting up Microsofts Application Center Test (ACT) and recording some vbscripts that could later be executed. Nowadays ACT is a lot sexier – but now it comes with Visual Studio 2010 but unfortunately only in Ultimate edition. I tried to persuade my wife to spend the $11000 on the ultimate edition – but she failed to see why this was more important than buying her a car.

Another good option is to use WebLoad. It’s a neat tool – and even if you buy it (to actually get a compiled and running version instead of the do-it-yourself-open-source) it still comes at a more decent price point. I recently played around with it – and it does solve a lot of your performance testing needs – but it’s almost a bit too much overkill for my need (which is essentially to find out how many request/s a web site can handle). I also didn’t like that it hijacked all my browsers and forced them to go through a proxy (in order for it to record what was going on) – and then failing to reset the proxy selection afterwards.

In the end I decided to spend the 30 min it would take to do a simple little performance tester of my own – that does exactly what I want it to.

I came up with AWebLoadTesting which is a compact and ultra-simple console app. It takes an input file which is essentially a text file with a list of urls to visit for each visitor during the test, an output filename – in which it will put a csv file with saved statistics – and that’s about it. If you need to you can also specify a hostname to run the test against – and even a custom UserAgent for the requests.

When it starts you have 0 visitors active. Then, by pressing “+” you can add visitors one at a time – and by pressing “1” and “5” you can add chunks of 10 or 50 visitors at a time. Each visitor is started in its own thread and will continuously go through the urls from the input file again and again.

“u” updates your view, “r” resets the counters", “s” saves the current data to the output file, “-“ removes a visitor” and of course “q” quits.

You’ll constantly be presented with the measured numbers: Time measured (s), Requests / s, Visitor count, Max load time, average load time and min. load time. On top of that it will show you a prioritized list of which urls are the slowest to return. That’s it.

The screenshot above is a test against a local EPiServer CMS 6.0 web site on my laptop, running with ASP.NET caching turned on (Set cache-expiration to 1h in episerver.config, site settings).

Download the binary here and the entire project here. Use AS-IS, LGPL 2.0, Quick&Dirty.

Guest Post: Конвертирование FB2 в PDF для Sony Reader PRS-505

Я - счастливый обладатель Sony Reader PRS-505!
Однако взявшись за чтение книг на русском языке, я обнаружила, что формат, в котором можно скачать большинство книг на русском в интернете FB2, не поддерживается моим Reader. Возникла проблема - как сконвертировать FB2 в один из форматов, поддерживаемых Sony Reader (например, PDF).
Найденная в интернете веб страничка http://fb2pdf.com/ , которая позволяет сконвертировать формат в PDF, к сожалению, сработала только в первый раз. В остальных случаях конвертация занимала слишком долгое время.
К счастью, мой муж - волшебник, и по совместительству программист, создал программу, которая позволяет сконвертировать FB2 в PDF в считанные секунды, и он безвозмедно предлагает ее всем любителям (электронных) книг на русском языке.
Загрузить программу можно здесь.

New Tool: Convert .FB2 files to PDF

It all started a couple of months ago when my wife, @othraen, got her Sony eReader. She loves it, and reads a couple of books a week on it. I can definitely see why she likes it – it’s slick, simple, and very pleasant to read on. Great toy, if you like books!

Anyway, with the speed she’s reading, not even the 5000000000000000 (or whatever count they reached now) free titles at Google Books is enough, and she’s started looking online for public domain Russian language books (most Russian is like Greek to me, but she likes it :-) ). As it turns out most of the Russian e-books online are in the FictionBook 2 format (.FB2). It’s an open XML format, probably defined by a league of Russian e-book publishers or so – it doesn’t seem to be all that popular in the rest of the world.

As you might imagine, the friendly people at Sony didn’t take this into consideration when planning on which formats to use in the eReader, so it is not supported. However, our friend, good old reliable PDFs are of course supported. It didn’t take many Google searches to find the fb2pdf.com website – a site devoted to converting fb2 files to PDF. It worked great on the first book….And on the second book. But later on, several of the books she tried to convert wouldn’t work. It would just freeze up and we could wait for ever and ever. As it so often happens, this resulted in the statement I fear so much “Honey, can’t you spend a little less time playing around with that programming stuff and a little more helping me get this book I want to read on my ereader??!” (yes, one of those questions where there’s only one right answer). In this case it would have made sense to simply drop a mail to the owners of the before mentioned website, asking them to fix whatever bug that caused the conversion to fail – but I was determined not to let this be the first time I give in to common sense :-)

Instead this was a perfect opportunity to take a closer look at FB2, PDFs, the iTextSharp library, SharpZipLib and ClickOnce deployment. At the same time, I’d get to program in my spare time – and with a perfect excuse – helping her :-)

It didn’t take much more than a couple of hours before I had a working windows application, that can load .fb2 files (or zip-files containing 1 .fb2 file), correct / change their Author / Title (since the ereader doesn’t seem to support Cyrillic characters in the title browsing menu), and convert into a PDF. It’s a ClickOnce application, which means you can install it directly from it’s online source – and it will automatically get updated if I upload a new version. The only tricky thing in developing it was actually embedding a font to show the Cyrillic Unicode characters that most Russian books are comprised of.

[UPDATE 2009-12-20] I finally found some time to fix a few bugs and add two new check-boxes: "Optimize for Sony Reader" and "Optimize for Kindle". Right now, all they do is to optimize page-sizes, so it should work better on the respective devices - but in the future they might also adjust font, font-sizes, etc. Let me know how well it works - I don't have a kindle, so I can't test it myself. Update should install itself.

Install the FB2 to PDF Converter tool from here.

Cool Tool: XDELTA

Yesterday, while trying to find a solution to a minor distribution problem I came across xdelta. It seems to be an awesome tool – just the way I like it: simple, fast, transparent and commandline based. It can find the difference between two binary files and store that compressed, and then later apply it as a patch. “Why is that neat?”, I hear you cry. Well – simple – now instead of storing a lot of really huge binary files where most of the contents is identical you can just store a source and then the different changes to that. Still don’t get it? Here’s the problem we’re going to solve with it.

Problem: X sales people located all around the world will get an external harddisk with a wmware image featuring all our amazing products for demo purposes. The problem arises whenever new versions of the image is created and needs to be distributed to all the sales people (probably going to happen several times a year). Not all internet connections are just as good for downloading 10gb files. Of course we could split it up, put it on an ftp-server or force everyone to install another of my favorite tools Free Download Manager. However, keeping our audience in mind, an automated solution would probably best.

Potential solution: On all the harddisks we put two images – a working copy they can use for demo’s and a virgin. Whenever we create a new version of the vmware image we’ll use xdelta to make a file containing all binary differences between the virgin image and the latest version – and at the same time compress the differences. We’ll do this using a command-line like this:

xdelta3.0u.x86-32.exe -e -5 -S djw -v -s "virgin-image-file" "newest-image-file" "difference-output-file"

The ‘-5’ indicates medium compression level, ‘-S djw’ is to turn on secondary compression, ‘-v’ is verbose mode – mostly because I’m a geek that likes to see what happens and ‘-s’ indicates the source file.
We can now let the sales people download the difference file together with a small batch-script that’ll apply the differences to their virgin-image and thereby recreating the latest version of the vmware image file using a syntax like this:

xdelta3.0u.x86-32.exe -d -s "virgin-image-file" "difference-file" "newest-image-file"

It tested it and it works. On a 9 GB image, I made some minor changes – like upgraded the EPiServer installed, removed old unused files, etc. That gave me a diff-file of 74 MB – which is a lot easier to download than 9 GB – and applying that file to a virgin image produced a new working image with the latest version installed :-)

Processing time on my laptop was around 800 seconds (< 15 min) for each action.

Enjoy

XSLT in EPiServer CMS 5

Personally I'm not a big fan of neither XSLT nor XML. In fact, my feelings around XML is expressed in this quote I heard recently: "XML is like children. They start out cute and small, then they grow..." (I don't remember who said it - if it was YOU, mail me and tell me to credit you for those words of wisdom). In my opinion XSLT's are mainly just good for job-security for XSLT developers - they are about as friendly to read as regular expressions - and terrible to maintain. Nevertheless a lot of people like them due to the way they help separate design from data - and I've already been asked the question "how can I work with XSLT in EPiServer" many times. So, now I thought I better do something about it, so one dark and cold evening I made a web control that hopefully will satisfy all the XSLT magicians out there!

The control will create a XML document representing the current page that has this structure:
    <page>
        <properties>
            <property name="PageLink" type="PageReference" isdynamic="False" isnull="False">3</property>
            ...
        </properties>
        <children>
            <page>
                ...
            </page>
        </children>
    </page>
and then transform that XML using the XSLT you provide.

In order to use the control, you'll need to place the dll in the "bin" folder, and register it on the page you wish to use it on. Then you can put it on the page like this:

<research:XSLT runat="server" id="xslt2" MaxChildDepth=1 IncludeDynamic="true"> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0" xmlns="http://www.w3.org/1999/xhtml"> <xsl:output method="html"/> <xsl:template match="page"> <h1><xsl:value-of select="properties/property[@name='PageName']"/></h1> <ul> <xsl:for-each select="properties/property"> <li><xsl:value-of select="@name"/>: <xsl:value-of select="."/></li> </xsl:for-each> <li> Children: <xsl:apply-templates select="children/page"/> </li> </ul> <br /> </xsl:template> </xsl:stylesheet></research:xslt>

In the above case the XSLT is specified within the controls tag, but you can also reference an external xslt file, by setting the property "TransformationFile" to the url of the XSLT file. Or - if you're feeling mean - you can bind the XSLT contents to a LongString property in EPiServer and let the editors figure out the extended stylesheets - that'll freak them out for sure :-)

By default the control will begin with rendering the XSLT on the current page data - if you want to base it on another page, it can be set up in well-known "PageLink" and "PageLinkProperty" properties.  The "IncludeDynamic" property specifies whether to include dynamic properties or not, and the "AutoHtmlDecode" property specifies if html-tags within properties should be rendered as HTML tags or as text on the page.

Find the control on labs.episerver.com

P.S. If you insist on playing around with XSLT and use this control, I wrote a XPathChecker some time ago that might come in handy.

Automatic Language Detection

A classical task when dealing with textual information is to automatically identify which language a text is written in (no, geeks - it's not a question of VB or C# - I mean human languages!).
Here's my attempt at a very simple, yet useful approach: character-bigram statistics.
I've basically made some extensive statistics on several languages on the frequency of all bigrams, and using that it's now possible to determine which language a given text resembles the most.
Try out my Language Detector here!

The text-corpus I used was another classic, the proceedings of the European Parlament through several years (can be found here).

My first step was to construct a class to contain bigram statistics for some text (LangStat).
In the class I also included code to determine the euclidean distance between two sets of bigram statistics (useful when trying to determine which language a text is most similar to). I implemented it as an operator overload for "-", so you can always determine the distance between two bigram-statistics by simply subtracting them from each other.

       //Calculates euclidean distance between two LangStat's
       public static double operator -(LangStat a,LangStat c)
       {
           //Operator overload
           double tot = 0;
           foreach (Bigram b in a.Bigrams.Keys)
           {
               if (c.Bigrams.ContainsKey(b))
               {
                   //Bigram exist in remote
                   double me = (double)a.Bigrams[b] / a.Count;
                   double them = (double)c.Bigrams[b] / c.Count;
                   tot += Math.Pow(Math.Abs(me - them), 2);
               }
           }
           return Math.Sqrt(tot);
       }

Then I build a Console trainer application, that is able to load the corpus text files for a given language, clean up any unwanted tags in them and then adds the text to a bigram statistic.

When it's done, it use the System.CodeDom to generate source-code for a class that inherits the LangStat, but which is specific to the current language. That way I'll have my languages precompiled and ready to be compared to custom textual content.
This might not be the most efficient approach, but it sure was funny to play around with CodeDom (an interesting namespace that I get to use far to seldom).

       static void Main(string[] args)
       {
           string lang = "sv";
           string langname = "Swedish";
           string[] files = Directory.GetFiles((...language folder...));
          
           //Build language statistics from file-corpus
           LangStat l=new LangStat();
           foreach(string f in files){
               Console.WriteLine("Examining file: "+f);
               StreamReader sr=new StreamReader(f);
               string s=sr.ReadToEnd();
               sr.Close();
               //File loaded
               s=Regex.Replace(s,"<[^>]*>"," ",RegexOptions.Multiline);
               //Tags removed
               l.AddText(s);
           }


           //Generate Code
           System.CodeDom.CodeNamespace ns =
               new System.CodeDom.CodeNamespace("Allan.Language.Detection");
           CodeTypeDeclaration tp = new CodeTypeDeclaration(langname);
           tp.BaseTypes.Add(typeof(LangStat));
           tp.IsClass = true;
           ns.Types.Add(tp);
           CodeConstructor cc = new CodeConstructor();
           cc.Attributes = MemberAttributes.Public;
           tp.Members.Add(cc);
           cc.BaseConstructorArgs.Add(
               new CodePrimitiveExpression(l.Bigrams.Count));
           foreach (Bigram b in l.Bigrams.Keys)
           {
               //Could be done much nicer, but I'm in a hurry
               cc.Statements.Add(
                   new CodeSnippetExpression(
                   "_bigrams.Add(new Bigram('"+b.A+"','"+b.B+"'),"+
                   l.Bigrams[b].ToString()+")"));
           }
           cc.Statements.Add(
               new CodeAssignStatement(
               new CodeVariableReferenceExpression("_count"),
               new CodePrimitiveExpression(l.Count)));
           System.CodeDom.Compiler.ICodeGenerator gen =
               new CSharpCodeProvider().CreateGenerator();
           StreamWriter sw=File.CreateText(langname+".cs");
           gen.GenerateCodeFromNamespace(ns, sw,
               new System.CodeDom.Compiler.CodeGeneratorOptions());
           sw.Close();

       }

Finally I just had to build a simple windows testing app, that will compare the text written to the languages. Download the solution here.

Generic Type Conversion in C#

I have now several times found myself in the peculiar situation of having two classes, that mostly have the same properties and fields, but doesn't implement the same interface or inherit from the same ancestor, which makes it rather tedious to convert between them.
The problem is typically seen when dynamically loading 3rd party libraries and trying to get them to interact with other 3rd party libraries, or when doing some advanced kinds of communication (like WCF) with complex types.
True, in some cases the problems can be avoided totally by considering your data, assembly and communication structure - but still there's those hopeless cases where you find yourself writing yet another "Create an object of this type based on that type"-code piece. Not a difficult task, but boring.
So, as always I've tried to come up with a stupid solution to a stupid problem: A Generic Type Converter. It uses Generics to "convert" type A to type B. By Converting it simply matches up the public properties and fields and copies the ones that match and that it's allowed to.
Sure, it might be slow (and some might even find it ugly) - but for tasks where the development time is more critical than a few miliseconds of execution time (like for POC's) it might be a nice thing to have in your toolbox.

Keep in mind that this is just a draft version 0.0.0.01.

P.S. Sorry about the formatting, but I'm having a small war with my blogspot and some CSS :-)

using System.Reflection;

public static class GenericTypeConverter
{

public static DestType ConvertType<SrcType, DestType>(SrcType Source) where DestType:class,new()
{
return ConvertType<SrcType,DestType>(Source, null, null);
}

public static DestType ConvertType<SrcType, DestType>(SrcType Source, Dictionary<string, string> SrcDestMapping) where DestType : class,new()
{
return ConvertType<SrcType,DestType>(Source, SrcDestMapping,null);
}

private static string GetMappedName(Dictionary<string, string> Map, string OrigName)
{
if((Map!=null)&&(Map.ContainsKey(OrigName))) return Map[OrigName];
return OrigName;
}

/// <summary>
/// Uses reflection to convert an object to a destination type, e.g. transfers all the properties and members they have in common
/// </summary>
/// <typeparam name="SrcType">Source Type</typeparam>
/// <typeparam name="DestType">Destination Type</typeparam>
/// <param name="Source">Object to convert</param>
/// <param name="SrcDestMap">Mapping between source and destination property names. Null if no mapping exist.</param>
/// <param name="Dest">Destination object or null if it should be created</param>
/// <returns>An object where as many properties and fields as possible have been transferred from Source.</returns>
private static DestType ConvertType<SrcType, DestType>(SrcType Source, Dictionary<string, string> SrcDestMap, DestType Dest) where DestType : class
{
//Create object if it doesn't exist.
DestType dstVar = Dest;
if (dstVar == null) dstVar = Activator.CreateInstance<DestType>();

//Loop through Source' public properties
Type srcTp = typeof(SrcType);
PropertyInfo[] props=srcTp.GetProperties(System.Reflection.BindingFlags.Public
| System.Reflection.BindingFlags.Instance
| System.Reflection.BindingFlags.Static
| System.Reflection.BindingFlags.GetProperty);
foreach (PropertyInfo p in props)
{
//Check if destination type has a settable property of the same type
PropertyInfo pDest = typeof(DestType).GetProperty(GetMappedName(SrcDestMap,p.Name), p.PropertyType);
if ((pDest != null) && (pDest.CanWrite)) pDest.SetValue(dstVar, p.GetValue(Source, null), null);
}

//Loop through Source' public fields
FieldInfo[] mems=srcTp.GetFields();
foreach (FieldInfo fi in mems)
{
FieldInfo mDest = typeof(DestType).GetField(GetMappedName(SrcDestMap,fi.Name));
if ((mDest != null) && (fi.FieldType == mDest.FieldType))
{
mDest.SetValue(dstVar, fi.GetValue(Source));
}
}

return dstVar;
}
}

Blog War: EPiServer vs Sitecore vs Reddot

Some months ago I came across Blogpulse.com (by Nielsen Buzzmetrics) which is a cool service that monitors how popular certain topics arein the Blog-sphere. Just for fun I ran this comparison between 3 of the major european CMS vendors: Sitecore, EPiServer and Reddot.
It would seem that all of the vendors have huge variations in their blog-popularity.

I also couldn't resist trying out battle-of-the-giants BlogPulse and this is what I came(Google vs. Microsoft vs. Apple) on the up with. It would seem that Google is in the lead Blog-wise, but both Microsoft and Apple are gaining (naturally you'd have to take into consideration that all blogs about the fruit that goes by the same name as a major soft/hardware company is also part of the graph).

ShotCode - Let's reinvent barcodes!

About a year and a half ago I was amazed at some of the new java applications for cameraphones I saw at the NeXT conference, among others were a reinvention of barcodes where the idea was that if a special 2d barcode was put on a poster, everybody with a cameraphone could scan it and get someinformation contained in the barcode, typically an ID number which an online service could translate to a URL or something else meaningful.
Of course I hurried home, downloaded the software and tried, but it almost never worked as it should so I gave up.

Today I found another similar service that actually seems to work quite well. And on top of storing your urls in a smart looking circular barcode they even offer to convert the webpage to a format more suitable for the phone that requests it (admitted, that specific part of their service doesn't seem to work too well, but anyway).

I saw the service in use at getjar.com - my favourity source for java apps for my nokia 6380 and there it worked quite impressive...Just start the shotcode application on the phone, take a picture of the barcode on the screen and begin downloading the java-application. Then I checked out Shotcode.com and after a quick registration I found it quite easy to create my own shotcodes....Now I suppose I'll just sit back and wait until this becomes an industry standard so everybody has it on their phones!

Another day another tool: XPath

Here's a small 5-minute tool I made the other day, simply because I needed it and it was faster to make it than finding a good one online that does what I want it to (although I'm sure hundreds of them exist).

It's a simple XPath tester, that can test an XPath against a piece of XML, or a URL pointing to XML.
It even attempts to do some simple syntax highlighting of the XML.

The tool is online at www.mizar.dk/XPath. (In case you are wondering, Mizar.dk is an old domain I currently use all my silly projects and for various other demo/lab purposes. It's actually a leftover from a webdesign company I had together with Jesper years ago).

The XPath tester can also be used for simply showing XML, and it accepts a couple of query-line parameters so you can send link to XML with applied XPaths like this.
The query-line parameters are: XML (raw xml for source) XMLURL (if you instead want to retrieve xml from a specific location) and finally XPATH (guess what thats for).

Enjoy!