Showing posts with label C#. Show all posts
Showing posts with label C#. Show all posts

Sunday, April 3, 2011

Cookieless Session State in ASP.NET without nasty URLs

Some of you have probably heard about the EU proposal that plans to end the internet as we know it on May 25th 2011. If you haven’t heard of it, David Naylor has made a nice little example of it’s consequences here. In essence most sites that use cookies will have to ask visitors to opt-in for every single cookie before using it. I’m very  much in favor of online privacy – yet it seems to me that this is a very poorly thought through directive. First of all, most cookies server 1 of 2 purposes:

  • Help web sites recognize visitors in order to provide them with the best possible service. Much like when I walk into my local barbershop and the barber recognizes me and knows exactly how I prefer him to cut my hair (the little I have left after reading crazy directives) – and which subjects I want to small talk about.
  • Visitor tracking in order to do statistics the site owners can use to improve the web site with. Again – it’s not all that different from when a grocery store owner thinks “wow – 10 customers this last week has asked me for low-fat milk. Perhaps I should start to carry that product here”.

I have no problem with both of the above scenarios – they fall into what I call good service and help enhance my online experience.
Another problem is that I generally dislike when legal stuff comes in the way for the best technical solution to a problem. Laws should describe the concept of what they are outlawing – not specific technical architectures such as cookies…But before I digress any further into political territory I’ll get right back on track.

Many ASP.NET developers rely on the Session State mechanism to store user relevant data within a visit, that can improve the user experience – for instance with personalization, prefilled forms, and so on. Unfortunately the Session state relies on a unique session key being stored in a local cookie in order to have a unique way to identify the same visitor throughout a visit. It actually comes with a built-in switch to make it stop using cookies – but unfortunately the solution looks rather ugly – it changes all the URLs on the site to contain a Guid and thereby track the visitor using the Guid. I, for one, am rather fond of clean and pretty friendly urls – so that’s no good. So – I started thinking…Many years ago I worked for a company that built a statistics tool. It was pretty unobtrusive and we didn’t use cookies. Instead we just tracked the source IP – and checked for repeated requests with a 10 minutes time-out. Sure, it wasn’t bullet-proof, but it actually worked surprisingly well. And in those cases where it didn’t work? Well – it was just 1 statistical entry out of many. It’s not like we used it to authorize access to the nuclear football, right?! Now, I thought that if we combine all the static information we get in the HTTP Request like IP, Accept Languages, Accept Types, User Agent and so on, smash it all together and take a fingerprint of it – we might end up with something that can almost be used as a session id. Consider: What are the odds that you’ll get 2 different visitors using the exact same configuration, coming from the exact IP on your site within the 20 minutes default time-out??
Of course it turns out I wasn’t the first to think this thought. In fact the clever people at the Electronic Frontier Foundation (EFF) has for some time been running a little example site that calculates those exact odds – just to prove that Privacy online isn’t solved by simply outlawing cookies.

So – I decided to put the thoughts into code. The code consist of 2 parts. First part is an extension method for the HttpRequest class, called “GetUniqueFingerprint()” which will return a MD5 Hash fingerprint.

using System;



using System.Collections.Generic;



using System.Linq;



using System.Web;



using System.Text;



using System.Security.Cryptography;



 



namespace AllanTech.NoCookie



{



    public static class NoCookies



    {



 



        static private string GetMd5Sum(string s)



        {



            Encoder enc = System.Text.Encoding.Unicode.GetEncoder();



            byte[] text = new byte[s.Length * 2];



            enc.GetBytes(s.ToCharArray(), 0, s.Length, text, 0, true);



            MD5 md5 = new MD5CryptoServiceProvider();



            byte[] result = md5.ComputeHash(text);



            StringBuilder sb = new StringBuilder();



            for (int i=0; i<result.Length; i++)



            {



                sb.Append(result[i].ToString("X2"));



            }



            return sb.ToString();



        }



 



        public static string GetUnqiueFingerprint(this HttpRequest Request)



        {



            string source=



                string.Join(",", Request.AcceptTypes)+";"+



                string.Join(",", Request.UserLanguages)+";"+



                Request.UserHostAddress+";"+



                Request.UserAgent;



            return GetMd5Sum(source);



        }



    }



}






Second part is a replacement for the ASP.NET SessionIDManager. This is the mechanism that uniquely identifies the visitor – either by a cookie or url – and by replacing it we can make it use our new UniqueFingerprint method instead. It’s really simple – just implement the ISessionIDManager and you’re good to go:





using System;



using System.Collections.Generic;



using System.Linq;



using System.Web;



using System.Web.SessionState;



 



namespace AllanTech.NoCookie



{



 



    public class CookielessIDManager : ISessionIDManager



    {



        public CookielessIDManager() { }



 



        #region ISessionIDManager Members



 



        public string CreateSessionID(HttpContext context)



        {



            return context.Request.GetUnqiueFingerprint();



        }



 



        public string GetSessionID(HttpContext context)



        {



            return context.Request.GetUnqiueFingerprint();



        }



 



        public void Initialize()



        {



            



        }



 



        public bool InitializeRequest(HttpContext context, bool suppressAutoDetectRedirect, out bool supportSessionIDReissue)



        {



            supportSessionIDReissue=true;



            return context.Response.IsRequestBeingRedirected;



        }



 



        public void RemoveSessionID(HttpContext context)



        {



        }



 



        public void SaveSessionID(HttpContext context, string id, out bool redirected, out bool cookieAdded)



        {



            redirected=false;



            cookieAdded=false;



        }



 



        public bool Validate(string id)



        {



            return true;



        }



 



        #endregion



    }



}




Finally, all I have to do is to change the configuration (web.config) to use my CookielessIDManager instead of the default:



<sessionState mode="InProc" sessionIDManagerType="AllanTech.NoCookie.CookielessIDManager,AllanTech.NoCookie" … /> 



Enjoy a site with 1 less cookie!

Thursday, September 3, 2009

HTTP Error 500.19 – Internal Server Error

A couple of times I have now run into this wonderful error after setting up a new EPiServer CMS site on a newly installed machine. However, I don’t think it’s specific to EPiServer CMS – but can occur in any asp.net application. The error message looks like this:

The requested page cannot be accessed because the related configuration data for the page is invalid.

As with many other microsoft errors, this is completely misleading (unless you’ve been messing up your configuration data – but this is a newly installed site).

The error message tries to clarify itself further:

Module
IIS Web Core

Notification
BeginRequest

Handler
Not yet determined

Error Code
0x80070021

Config Error
This configuration section cannot be used at this path. This happens when the section is locked at a parent level. Locking is either by default (overrideModeDefault="Deny"), or set explicitly by a location tag with overrideMode="Deny" or the legacy allowOverride="false".

Config File
\\?\C:\EPiServer\Sites\MyEPiServerSite3\web.config

 

By now, you are probably half way into your web.config, and trying to remember where that mystical machine.config is located. You might even have started to go through all directories on your machine in the search for another web.config thats somehome conflicting. Stop looking. It’s not conflicting. And relax – you didn’t do anything wrong…yet.

One of 2 things have likely happened:

1) IIS was installed after ASP.NET, and it’s not really sure what to do with ASP.NET code. Solution: In a command prompt run aspnet_regiis. It’s typically found in “%windows%\Microsoft.Net\Framework\v2.0.50727\aspnet_regiis –i”. Note that you need to run it as Administrator on Vista and Windows 7. In fact, running this can be a cure for many initial web server errors on newly installed asp.net applications.

2) You are running Windows 7 or Windows Server 2008 and you selected the default install of IIS7.5. For reasons known only to Microsoft, they decided that ASP.NET shouldn’t be included in a default installation. My personal theory is that the guy who made that wise call is probably the same guy who decided to remove the ability to drag files onto a command-prompt in Vista. And before that was the mastermind behind the “Copy File” feature in any earlier version of Windows. But I digress :-) Solution in this case is easy – Open Control Panel | Programs | Turn Windows Features on and off, find IIS and enable ASP.NET:

image

Hit “Ok”, wait for half an hour and then see your web application spring to life. Enjoy!

Sunday, August 2, 2009

New Tool: Convert .FB2 files to PDF

ereader

It all started a couple of months ago when my wife, @othraen, got her Sony eReader. She loves it, and reads a couple of books a week on it. I can definitely see why she likes it – it’s slick, simple, and very pleasant to read on. Great toy, if you like books!

Anyway, with the speed she’s reading, not even the 5000000000000000 (or whatever count they reached now) free titles at Google Books is enough, and she’s started looking online for public domain Russian language books (most Russian is like Greek to me, but she likes it :-) ). As it turns out most of the Russian e-books online are in the FictionBook 2 format (.FB2). It’s an open XML format, probably defined by a league of Russian e-book publishers or so – it doesn’t seem to be all that popular in the rest of the world.

As you might imagine, the friendly people at Sony didn’t take this into consideration when planning on which formats to use in the eReader, so it is not supported. However, our friend, good old reliable PDFs are of course supported. It didn’t take many Google searches to find the fb2pdf.com website – a site devoted to converting fb2 files to PDF. It worked great on the first book….And on the second book. But later on, several of the books she tried to convert wouldn’t work. It would just freeze up and we could wait for ever and ever. As it so often happens, this resulted in the statement I fear so much “Honey, can’t you spend a little less time playing around with that programming stuff and a little more helping me get this book I want to read on my ereader??!” (yes, one of those questions where there’s only one right answer). In this case it would have made sense to simply drop a mail to the owners of the before mentioned website, asking them to fix whatever bug that caused the conversion to fail – but I was determined not to let this be the first time I give in to common sense :-)

image Instead this was a perfect opportunity to take a closer look at FB2, PDFs, the iTextSharp library, SharpZipLib and ClickOnce deployment. At the same time, I’d get to program in my spare time – and with a perfect excuse – helping her :-)

It didn’t take much more than a couple of hours before I had a working windows application, that can load .fb2 files (or zip-files containing 1 .fb2 file), correct / change their Author / Title (since the ereader doesn’t seem to support Cyrillic characters in the title browsing menu), and convert into a PDF. It’s a ClickOnce application, which means you can install it directly from it’s online source – and it will automatically get updated if I upload a new version. The only tricky thing in developing it was actually embedding a font to show the Cyrillic Unicode characters that most Russian books are comprised of.

[UPDATE 2009-12-20] I finally found some time to fix a few bugs and add two new check-boxes: "Optimize for Sony Reader" and "Optimize for Kindle". Right now, all they do is to optimize page-sizes, so it should work better on the respective devices - but in the future they might also adjust font, font-sizes, etc. Let me know how well it works - I don't have a kindle, so I can't test it myself. Update should install itself.

Install the FB2 to PDF Converter tool from here.

Sunday, January 4, 2009

Fun with card games

I just published one of my holiday pet projects on Codeplex. It's a very basic framework for building your own card games in .net.

With time, I'll also publish some sample projects, and hopefully even expand the project to include some basic AI logic as well.

Check it out: http://www.codeplex.com/CardGameLib

Even though it's still missing a lot of parts, I've already begun to use it for 2 other pet projects. One is a code competition I plan to put out soon - just looking for price-sponsors now, the other is a web-edition of the classic 500-Rummy card game, a game that I really enjoyed growing up - and still play when I'm vacationing with my family. Card games are an excellent form of entertainment - and remember, there's more than poker and blackjack to life.

Drop me a line - or leave a comment, if you want to contribute to the project, sponsor a price for the upcoming code-competition or just have an opinion to share :-)

Thursday, February 14, 2008

Nightly Fun with 301

WARNING: GEEKY STUFF.

 

Do you ever have difficulty falling asleep at night, because your brain begins to code the moment your tired body hits the bed? Well, I do every once in a while. And then I know that I can either toss and turn all night or sneak downstairs and code for a while - just to get it out of my system.

Last night was one of those nights. This time, though, I decided to make something simple but useful. The idea had been in the back of my head ever since I visited a large client recently who was getting ready to move a big and popular web site in the media industry to EPiServer CMS 5.

Basically this is the classic problem that I'm trying to solve: When moving a web site to a new platform it often happens that the url to the individual pages changes which breaks links, search engine ranks and results in a lot of 404 errors. For instance, with Friendly URLS in EPiServer a page that used to be called "/articles/article124145.html" might now get the friendlier path"/News/New+Website+Launched/". So, my solution to this problem is simply a generic HttpModule that loads a series of old-path / new-path sets from a text-file when the web application is started and then makes a quick check for every request to see if it's actually an old URL being requested. If it is, it'll do the correct thing and send back an HTTP 301 Permanently Moved reply with the new location of the document. That way both global search engines and various caches should get updated and no links will be broken. Simple, but it seems to work - and to my luck it was so easy to code that I still got most of a good nights sleep.

 

    public class RedirectModule : IHttpModule
{
private Dictionary<string, string> map;

public void Init(HttpApplication app)
{
app.BeginRequest += new EventHandler(app_BeginRequest);
map = new Dictionary<string, string>();
//Load from file
StreamReader sr = File.OpenText(ConfigurationManager.AppSettings["UrlMapping"]);
string s = null;
while ((s = sr.ReadLine()) != null)
{
string[] parts = s.Split('|');
if(parts.Length==2) map.Add(parts[0], parts[1]);
}
sr.Close();
}

void app_BeginRequest(object sender, EventArgs e)
{
HttpApplication app = (sender as HttpApplication);
if (map.ContainsKey(app.Request.Url.LocalPath))
{
app.Response.Status = "301 Moved Permanently";
app.Response.AddHeader("Location", map[app.Request.Url.LocalPath]);
app.Response.End();
}
}

public void Dispose()
{
}
}



 



As said - it didn't keep me up all night.



In order for it to work the following adjustments needs to go into web.config:



<appSettings>

  <add key="UrlMapping" value="c:\\inetpub\\UrlMapping.txt"/>


</appSettings>



<system.web>

  <httpModules>


    <add name="RedirectModule" type="EPiServer.Research.RedirectModule.RedirectModule,EPiServer.Research.RedirectModule"/>


  </httpModules> 
</system.web>



 



And the mapping file should just be a series of [path]|[new path], one on each line like this:



/oldpath/article.html|/newpath/newarticle.aspx

/oldpath/oldpage|http://www.google.com/?q=oldpage+topic

Friday, November 23, 2007

XSLT in EPiServer CMS 5

Personally I'm not a big fan of neither XSLT nor XML. In fact, my feelings around XML is expressed in this quote I heard recently: "XML is like children. They start out cute and small, then they grow..." (I don't remember who said it - if it was YOU, mail me and tell me to credit you for those words of wisdom). In my opinion XSLT's are mainly just good for job-security for XSLT developers - they are about as friendly to read as regular expressions - and terrible to maintain. Nevertheless a lot of people like them due to the way they help separate design from data - and I've already been asked the question "how can I work with XSLT in EPiServer" many times. So, now I thought I better do something about it, so one dark and cold evening I made a web control that hopefully will satisfy all the XSLT magicians out there!

The control will create a XML document representing the current page that has this structure:
    <page>
        <properties>
            <property name="PageLink" type="PageReference" isdynamic="False" isnull="False">3</property>
            ...
        </properties>
        <children>
            <page>
                ...
            </page>
        </children>
    </page>
and then transform that XML using the XSLT you provide.

In order to use the control, you'll need to place the dll in the "bin" folder, and register it on the page you wish to use it on. Then you can put it on the page like this:

<research:XSLT runat="server" id="xslt2" MaxChildDepth=1 IncludeDynamic="true"> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0" xmlns="http://www.w3.org/1999/xhtml"> <xsl:output method="html"/> <xsl:template match="page"> <h1><xsl:value-of select="properties/property[@name='PageName']"/></h1> <ul> <xsl:for-each select="properties/property"> <li><xsl:value-of select="@name"/>: <xsl:value-of select="."/></li> </xsl:for-each> <li> Children: <xsl:apply-templates select="children/page"/> </li> </ul> <br /> </xsl:template> </xsl:stylesheet></research:xslt>





In the above case the XSLT is specified within the controls tag, but you can also reference an external xslt file, by setting the property "TransformationFile" to the url of the XSLT file. Or - if you're feeling mean - you can bind the XSLT contents to a LongString property in EPiServer and let the editors figure out the extended stylesheets - that'll freak them out for sure :-)



By default the control will begin with rendering the XSLT on the current page data - if you want to base it on another page, it can be set up in well-known "PageLink" and "PageLinkProperty" properties.  The "IncludeDynamic" property specifies whether to include dynamic properties or not, and the "AutoHtmlDecode" property specifies if html-tags within properties should be rendered as HTML tags or as text on the page.



Find the control on labs.episerver.com



P.S. If you insist on playing around with XSLT and use this control, I wrote a XPathChecker some time ago that might come in handy.

Friday, November 9, 2007

A Simple Page Import Web Service

In EPiServer CMS 5 there's a couple of very useful Web Services that gives you pretty thorough access to do just about anything you please - at least with regards to adding / searching / modifying pages.

However the Web Services can be pretty complex and at times a green newbie like myself needs a quick & dirty way of importing files from another system into EPiServer without doing too much thinking. In fact, I found myself in exactly that situation recently, when I wanted to import a huge amount of test data I had scavenged on the net (19040 pages to be exact). So I ended up writing a new little web service that takes the following parameters:

  • ParentID - The ID of the page that should be the parent of the page you're adding
  • PageType - The name of the page type to create
  • PageXML - A string of XML, defining the page.The tag-names should match the page properties you want to set, and they should all be wrapped in a <page> </page> tag. So something like "<page><pagename>My Page</pagename><bodyField>This is the body</bodyField></page>"
  • Publish - A boolean parameter that specifies if the page should be published instantly.

The Web method then returns the page-id of the page created, thereby enabling you to build your own page-hierarchy.

In order to make it, I simply created a new standard Web Service in Visual Studio, set the following Using clauses:

using EPiServer;
using EPiServer.DataAccess;
using EPiServer.Core;
using EPiServer.DataAbstraction;
using System.Xml;
using EPiServer.Security;

and added this method to the Web Service class:

[WebMethod]
public int ImportPage(int ParentID, string PageType, string PageXML, bool Publish)
{
    XmlDocument xd = new XmlDocument();
    xd.LoadXml(PageXML);
    PageData pd = DataFactory.Instance.GetDefaultPageData(new PageReference(ParentID), PageType,AccessLevel.NoAccess);
    //Fill in properties
    foreach (XmlNode xn in xd.DocumentElement.ChildNodes)
    {
        if (xn is XmlElement)
        {
            string name = (xn as XmlElement).Name;
            try
            {
                pd[name] = (xn as XmlElement).InnerText;
            }
            catch { }
        }
    }

    PageReference pr=DataFactory.Instance.Save(pd, (Publish)? SaveAction.Publish:SaveAction.Save,AccessLevel.NoAccess);
    return pr.ID;
}

 

So, basically it creates a page with default data, iterates through the 1st level nodes and checks if there is a matching page property. This is a really simple example without any proper error handling, and which only supports string properties. The only thing that's worth noting is how I use the AccessLevel.NoAccess in the Save method and the GetDefaultPageData method, to avoid uncomfortable access checks (since the web service typically runs as an anonymous user). However, make sure always to put the service behind access-control (which can be set up in web.config).

 

And yes, the import of my 19k pages went surprisingly well - and pretty quick too!

Monday, October 22, 2007

How to extend the general Page functionality in EPiServer CMS 5

Still being a newbie in the EPiServer company I learn a lot of new things every day - both about the product and the company.

I've decided to share some of my discoveries here - perhaps they help other EPiServer newbies out there :-)

Today I came across quite a useful, but not-so-well-documented feature in EPiServer CMS 5: The ability to extend the general functionality of your Pages. So, if for instance you want to add a piece of code that should be executed whenever a page is shown, and which can affect that page, but don't feel like inheriting the TemplatePage type and letting all your pages inherit from your custom type, there is actually quite a neat way of doing it.

The trick is the PagePlugIn attribute. You can create a class, put the PagePlugIn attribute (from the EPiServer.PlugIn namespace) and add static method with the signature void Initialize(int) and from that method setup an event handler to handle the EPiServer.PageBase.PageSetup event.

Here's an example on how you can use this functionality to add an extra menu item to the context menu:





[PagePlugIn]
public class MyPagePlugin
{

public static void Initialize(int bitflags)
{
EPiServer.PageBase.PageSetup += new EPiServer.PageSetupEventHandler(PageBase_PageSetup);
}

static void PageBase_PageSetup(EPiServer.PageBase sender, EPiServer.PageSetupEventArgs e)
{
sender.ClientScript.RegisterClientScriptInclude("OnScript", "MyScript.js");
sender.PreRender += new EventHandler(sender_PreRender);
}

static void sender_PreRender(object sender, EventArgs e)
{
(sender as EPiServer.PageBase).ContextMenu.Menu.Add("MyItem", EPiServer.Security.AccessLevel.Edit, new EPiServer.RightClickMenuItem("My Script", "MyScript()", "MyScriptSubMenu"));
}

}

Monday, September 24, 2007

EPiServer Code: Send a warning email when a page is about to expire

As you might imagine I find it difficult to keep my hands of the brand new v.5. So naturally I've been searching out excuses to try out coding some small samples against the API.
Here's a feature that I've heard requested from several intranet customers already - An automatic email that informs the owner of a page that it's about to expire.
"What a great idea" I thought first time I heard it - One of the major problems with most intranet is outdated information so I can easily imagine companies having a policy that all pages on their intranet should have an expiration date - and I can just as easily imagine the need for owners to change that expiration date, if the information is still relevant hence producing the need for a warning email.
First off I could see several approaches to making this in EPiServer:
  1. Hook into the right DataFactory event from the global.asax and send a mail whenever a page is expiring
  2. Use the v5 support for workflow foundation and make a workflow that performs a SendEmail activity when a page is moved to the archive
  3. Set up a scheduled task to check for pages about to expire
Although I found option 2 very encharming due to the use of workflows, I decided to make an implementation of option 3 - since this was the only approach that would send out a warning email before the page actually expired (and any damage was done).

I made a new C# Code library project, added references to the relevant EPiServer dlls (and log4net to enable logging) and wrote this code:



using System;
using System.Collections.Generic;
using System.Text;
using EPiServer.PlugIn;
using EPiServer;
using EPiServer.Core;
using EPiServer.Filters;
using EPiServer.Security;
using EPiServer.Personalization;
using EPiServer.Configuration;
using System.Net.Mail;

namespace Allan.EPiModules
{
[ScheduledPlugIn(DisplayName = "Page Expiry Warning")]
public class ExpiryWarningJob
{
private static log4net.ILog _log;

static ExpiryWarningJob()
{
_log = log4net.LogManager.GetLogger(typeof(ExpiryWarningJob));
}

public static string Execute(){
int num = 0;
//Find pages that will expire in a day
PropertyCriteria criteria = new PropertyCriteria();
criteria.Name = "PageStopPublish";
criteria.Value = DateTime.Now.AddDays(1).ToString();
criteria.Type = PropertyDataType.Date;
criteria.Required = true;
criteria.Condition = CompareCondition.LessThan;
//...but hasn't already expired
PropertyCriteria criteria2 = new PropertyCriteria();
criteria2.Name = "PageStopPublish";
criteria2.Value = DateTime.Now.ToString();
criteria2.Type = PropertyDataType.Date;
criteria2.Required = true;
criteria2.Condition = CompareCondition.GreaterThan;
PropertyCriteriaCollection criterias = new PropertyCriteriaCollection();
criterias.Add(criteria);
criterias.Add(criteria2);
foreach (PageData data in DataFactory.Instance.FindPagesWithCriteria(
PageReference.RootPage, criterias,
null, LanguageSelector.MasterLanguage(), AccessLevel.NoAccess)
)
{
SendMail(data);
num++;

}
return string.Format("{0} expiry emails sent", num.ToString());

}


private static void SendMail(PageData p)
{
//Identify user profile.
//Consider using the ChangedBy instead of CreatedBy.
EPiServerProfile esp = EPiServerProfile.Get(p.CreatedBy);
if (esp.Email != null)
{
try
{
//Build a new mail message
MailMessage message = new MailMessage("expire@" + Settings.Instance.SiteUrl.Host, esp.Email);
message.Subject = "Page \"" + p.PageName + "\" is about to expire";
message.Headers.Add("X-Mailer", "EPiServer CMS");
message.Headers.Add("Content-Base", Settings.Instance.SiteUrl.GetLeftPart(UriPartial.Authority));
message.Body = "The page <A href=\""
+ Settings.Instance.SiteUrl.GetLeftPart(UriPartial.Authority)
+ p.StaticLinkURL + "\">"
+ p.PageName
+ "</A> will expire on "
+ p.StopPublish.ToShortDateString();
message.IsBodyHtml = true;
message.BodyEncoding = Encoding.UTF8;
SmtpClient smtp = new SmtpClient();
//Make sure the web.config sets up the SMTP Client.
smtp.Send(message);
_log.Info("Expiry warning sent to: " + esp.Email);
}
catch (Exception e)
{
_log.Error("Failed to send expiry warning", e);
}
}
else _log.Warn("Unable to send expiry warning to " +
esp.DisplayName +
" - no known email address");
}
}
}

The dll should be placed in the EPiServers "bin" folder and then it will automatically be loaded. The ScheduledPlugin attribute will make it appear as a scheduled task in Admin mode. Here you should probably set it to run once a day - perhaps in the early hours of the morning will be best.
You also need to make sure the web.config is setup to the SMTP server.

The code is a pretty simple sample that will find the pages that are about to expire (the following day) and then send a mail to the creator of each page that it's about to expire.

The sample was made in less than a day and of course it still could use a lot of work to be really nice. Ideas for improvements:
  • Group expiration mails so that each user won't be bombarded with several mails every day
  • Consider how long time before people should be warned that the pages are about to expire - is 1 day time enough?
  • Consider if it's really the creator that should get the mail - perhaps the last person to have updated the page would be the right one?
  • Build functionality together with tasks - so instead of emails a task to check the page should be created.
Enjoy!

Friday, July 13, 2007

Automatic Language Detection

A classical task when dealing with textual information is to automatically identify which language a text is written in (no, geeks - it's not a question of VB or C# - I mean human languages!).
Here's my attempt at a very simple, yet useful approach: character-bigram statistics.
I've basically made some extensive statistics on several languages on the frequency of all bigrams, and using that it's now possible to determine which language a given text resembles the most.
Try out my Language Detector here!

The text-corpus I used was another classic, the proceedings of the European Parlament through several years (can be found here).

My first step was to construct a class to contain bigram statistics for some text (LangStat).
In the class I also included code to determine the euclidean distance between two sets of bigram statistics (useful when trying to determine which language a text is most similar to). I implemented it as an operator overload for "-", so you can always determine the distance between two bigram-statistics by simply subtracting them from each other.



//Calculates euclidean distance between two LangStat's
public static double operator -(LangStat a,LangStat c)
{
//Operator overload
double tot = 0;
foreach (Bigram b in a.Bigrams.Keys)
{
if (c.Bigrams.ContainsKey(b))
{
//Bigram exist in remote
double me = (double)a.Bigrams[b] / a.Count;
double them = (double)c.Bigrams[b] / c.Count;
tot += Math.Pow(Math.Abs(me - them), 2);
}
}
return Math.Sqrt(tot);
}


Then I build a Console trainer application, that is able to load the corpus text files for a given language, clean up any unwanted tags in them and then adds the text to a bigram statistic.

When it's done, it use the System.CodeDom to generate source-code for a class that inherits the LangStat, but which is specific to the current language. That way I'll have my languages precompiled and ready to be compared to custom textual content.
This might not be the most efficient approach, but it sure was funny to play around with CodeDom (an interesting namespace that I get to use far to seldom).



static void Main(string[] args)
{
string lang = "sv";
string langname = "Swedish";
string[] files = Directory.GetFiles((...language folder...));

//Build language statistics from file-corpus
LangStat l=new LangStat();
foreach(string f in files){
Console.WriteLine("Examining file: "+f);
StreamReader sr=new StreamReader(f);
string s=sr.ReadToEnd();
sr.Close();
//File loaded
s=Regex.Replace(s,"<[^>]*>"," ",RegexOptions.Multiline);
//Tags removed
l.AddText(s);
}


//Generate Code
System.CodeDom.CodeNamespace ns =
new System.CodeDom.CodeNamespace("Allan.Language.Detection");
CodeTypeDeclaration tp = new CodeTypeDeclaration(langname);
tp.BaseTypes.Add(typeof(LangStat));
tp.IsClass = true;
ns.Types.Add(tp);
CodeConstructor cc = new CodeConstructor();
cc.Attributes = MemberAttributes.Public;
tp.Members.Add(cc);
cc.BaseConstructorArgs.Add(
new CodePrimitiveExpression(l.Bigrams.Count));
foreach (Bigram b in l.Bigrams.Keys)
{
//Could be done much nicer, but I'm in a hurry
cc.Statements.Add(
new CodeSnippetExpression(
"_bigrams.Add(new Bigram('"+b.A+"','"+b.B+"'),"+
l.Bigrams[b].ToString()+")"));
}
cc.Statements.Add(
new CodeAssignStatement(
new CodeVariableReferenceExpression("_count"),
new CodePrimitiveExpression(l.Count)));
System.CodeDom.Compiler.ICodeGenerator gen =
new CSharpCodeProvider().CreateGenerator();
StreamWriter sw=File.CreateText(langname+".cs");
gen.GenerateCodeFromNamespace(ns, sw,
new System.CodeDom.Compiler.CodeGeneratorOptions());
sw.Close();

}


Finally I just had to build a simple windows testing app, that will compare the text written to the languages. Download the solution here.

Wednesday, July 11, 2007

Majestic

A major problem for most global search engines is the simple fact that the net grows so rapidly that no matter how many serverfarms they build, pages are being created or updated faster than the search engines can detect and index them.
I recently came across Majestic that has a really interesting approach to this problem: Distributed crawlers. They've made a simple crawler-client that can help distribute the indexing among all the volunteers who provide spare bandwidth and computertime to this noble task in much the same way as some people donate time and bandwidth to the SETI@HOME project or my personal favourite, the search for the next Mersenne prime.
However, the idea with distributing the search seems really useful. Now, if only they had done something novel to the search-end instead of just copying Google I would have been thrilled. But I like the idea anyway. Check it out at http://www.majestic12.co.uk

Oh yeah, while you're there, check out the C# source for their HTML Parser. It's awesome. Fast and furious!