Migrating mails from IMAP to IMAP (part 1/2)

One of my clients wanted to change its hosting provider because of almost constant downtime. I suggested that we move the whole site and the associated email addresses to Dreamhost.

He is heavily using its webmail, with a dozen of different accounts, all with custom address book and drafts to reuse.

I had already moved a previous site of the same client to a GMail powered webmail and he wasn't so satisfied of it because of the lack of a "draft" feature that would allow him to reuse the same message templates.

Its current webmail is powered by Imp (Horde) and the default Dreamhost webmail is Squirel Mail.

So now I'm faced with a challenge. Which webmail should I choose and how will I initiate the migration ?

Contestant 1 : Horde

The easiest way would have been to reuse Horde. Horde has a nifty export/import function. It means that I could have exported the settings on the current host and imported them back in the new host.

Unfortunatly, Horde is not pre-installed on Dreamhost. It means I would have to install it and I don't really want to do it. This is not my area of expertise and if something goes wrong it will take me ages to fix.

Additionnaly, the default php5 install on Dreamhost lacks the IMAP extension needed and SSL connection requires an additional monthly fee.

That would be the better solution for the client, but I don't feel capable of doing it.

Contestant 2 : Squirel Mail

Squirel Mail is the default webmail on Dreamhost. It means that they do handle all the heavy stuff of configuring it and making sure it does not crash.

They assure me that I can have unlimited hosting of my mails, the only limits are a maximum of 100 SMTP sending per hour and no attached filed > 7Mo.

Those are almost no issues at all, we can handle that. I just have to check if the 100 sending per hour is for each mail address or for the entire account.

The only problem will be that I can't export/import the horde feature into Squirel.

What I could do however is synchronize the two IMAP servers, effectively copying all the mail from the current one to the new. Unfortunatly, there is no easy way to do that.

The Dreamhost support even told me that it was impossible. I found a linux command named imapsynch however that maybe could help, but I'll need to know the name of the dreamhost imap server and needed credentials to connect to it. I'll sort this out with them.

Contestant 3 : GMail

I've already set mails up with GMail a couple of times in the past so this won't be too hard. Could take some times, but nothing difficult about that.

I may be able to copy the current mails to the GMail server, I've seen some articles on the subject floating on the network, but I'll definitly won't be able to copy drafts.

Conclusion

I'm still in contact with the Dreamhost support to choose the better option. I don't think I'll install Horde, this will be too much work. I guess I'll go with the second option of migrating mails to Squirel Mail.

Whatever solution I'll pick, I'll be sure to post the whole details here for others to follow.

Update

I finally won't have to migrate mails, the client makes regular POP backup of its mails so he will just do one more right before the transfer and will start over with a brand new mail boxes. Concerning the drafts, he will manually copy-paste them from one webmail to the other and he told me that he wasn't using the address book...

So it appears that I'll just have to change the DNS to point to the new webmail and create a temporary CNAME to still access the old webmail on the old server until the migration is totally over.

Re-update

In fact, it appears that I won't be able to access both webmail at the same time. I'll have DNS issues, and the registrar panel that I have is very basic, I won't be able to fine tuned it correctly.

It seems that I'll have to migrate mails from one server to the other.

Pros and cons of using Google CDN for your assets

With the recent Google Fonts breakout, much debate surfaces to know if it really was useful to use Google CDN to deliver your assets to your users.

I used to think it was a goog thing because it allowed for parallelized download, but after reading really good articles on the subject I'm not so sure.

I'll try to list the pros and cons of using Google CDN instead of serving your files locally.

Pros

• Using a CDN assure you that your users will get content from the closest server around them, no matter where they are.
• The more websites will use Google CDN, the biggest the chance of your user already having the file in its cache
• Downloading from a CDN assure that you won't be sending useless cookies along the request. You shouldn't send cookies when fetching your static files anyway, but sometime -Google Analytics for example-, some script create domain-wide cookies that still get sent.
• Many niched major websites (ArmorGames.com, StackOverflow.com, etc) uses the CDN, so if a large part of your trafic is coming from one of those, using the CDN and the exact same library version will have no perfomance downsides for you.

Cons

• Using a CDN create a new connection to access the ressource on an other server instead of using the already open connection to your website. Creating a new connection is slow.
• Caching of the files will only be possible if the exact same version url is used. E. using /jquery/1.4.0/ instead of /jquery/1.4/
• If you already are using your own CDN for your images, you should also use it for your javascript and in this case the google one is worthless because you already have all the advantages.
• An Internet connection is required. This may seem obvious but if you have to work on a project while on a trip, you'll need a local copy of the library.

Stalemate

• Using a CDN force a DNS lookup. This lookup can be cached if the user already visited your website or another that use the same CDN however.
• Browsers have a limit of paralels downloads on the same host. Using a CDN allow for more parallelization. But creating dummy subdomains like static.domain.com would also defeat this limitation.
• In the case of Javascript/CSS, packing all your files (scripts and libraries) in one big file on your server and serving it gzipped will result in a better compression that serving the library from a CDN and your scripts from your server. On the other hand, the user would have to re-download the whole package whenever you make a tiny change to your scripts, even if you didn't changed the library.
• Downloading script usually block page execution. So reducing the number of scripts to load (by packing them in one file instead of downloading one locally and one on the CDN) will reduce the blocking time. On the other hand, merging two files will result in a longer download time (because the file is larger) while using the CDN would allow the two scripts to download in parallel.
• What happen if your CDN is down ? Well, when using Google that's not much a risk but still. As a side effect, you should'nt even consider using a CDN if your application is for an intranet.

Conclusion

As a conclusion, all I can answer to the "Should I use the Google CDN for my javascript libraries ?" is "Well... it depends...".

Really it depends on your configuration. Run some tests, try with a CDN or by compressing the library into your main file. Check where your users are coming from and the average loading time. Test various configuration until you found the one that works the best for you.

On a personal note I do use Google CDN for my websites. The cost of the DNS lookup and connection to the Google server is really worth the parallelized download in my case. But it may be so because I'm living in France and my website is hosted in the US, so a CDN really helps.

Source links

• http://docs.google.com/document/pub?id=14Xn7kkKwVxSie6xHJwWRAP5mutM7qlWcrHgWtZo63gI
• http://zoompf.com/blog/2010/01/should-you-use-javascript-library-cdns

Doing persistent login the right way

When I was a young web developer, I had to code an admin panel, with username and login. I also added the well-known "Remember me" checkbox.

I wasn't really educated to the security concerns of developing such a feature. In fact, I was even storing the passwords as plain text in the database.

And to allow the user to auto-login, I stored the user and password, in plain text, in a cookie and requested a login with those credentials whenever its session expired.

That's insane !

Yes it was, but I remember thinking "Well, nobody can even have access to the database so there's no risk" and "I can store the password in the cookie, it is only accessible by the user, and he surely already know its password. There's no harm in that"

Well, XSS and CSRF weren't so common back in those days.

Since then I've learned a lot. There's just no way I'll ever store a plain text or uncrypted password in a database again. Hashing and salting is the way to go.

But I kept implementing the persistent login feature following the same pattern : saving the username and hashed password in the cookie.

I was thinking "The password is not stored as plain text but as a hash, there's no way an attacker would even be able to guess the password from that".

That's better, but still...

I haven't thought that the cookie was like an open sesame to my app. One would just have to steal the file and place it in its browser cookie list and he would have exactly the same access to the website, without having to know my password.

In the meantime I was also implementing OpenId authentication and as it is passwordless (at least from the website point of view), I couldn't save the username and password in the cookie.

Clearly, something was broken in my persistent login implementation.

Here come the tokens

After some research, I finally get it the right way.

Whenever a user authenticate, be it from a classical user/pass login form or using OpenId, I keep it loggued using session vars. When the session var expires, the user is loggued out. If he had checked the "Remember me" checkbox, I would try to re-authenticate him automatically.

In order to do that, I had to change my implementation. The first thing to do was creating a UserTokentable with 3 fields : user_id, tokenand duration.

Then, when the user authenticate I generate a random token and save it along the user_id in a cookie.

I'll also populate the UserTokentable with the user_id, a hashed version of the token,  and matching the durationto the expires date of the cookie.

That way, the next time that the user comes to my website without a session I check its cookie. If he has a user_id and a token, I'll try to authenticate him.

I'll find a matching user_id and hashed token. If there are no results, then the cookie is not valid. Either it has been already used or it is a cookie forgery attempt. In that case, I'll do nothing.

If there's a match, I still check that the token is not expired. If it is, I'll delete it in the database, as well as clearing the cookie. On the other hand, if the cookie is still valid, I log the user in.

Once the user is loggued in I generate a new token, update the cookie and update the table with a hashed version of it.

That way each token can only be used once and are generated only when the user correctly authenticate and are only used once the session expires.

But one can still steal cookies

Yes, a malicious user can still steal my cookie, copy it to its computer and use the token to login. That's true and as we can't do anything against that, we could still try to mitigate it.

First precaution would be to disable any sensitive information edition while loggued in using the cookie. Like bank account, personnal information, password changing, etc.

The second thing would be to check for any hacking attempt. Whenever I guess for a forgery attempt (trying to login with a bad set of id / hashed token), I delete all UserToken associated for this user. This will block any bruteforce attack because even there won't be any more match in the database.

It is also wise to display a warning message to the user telling him that his account may have been compromised and that he should be wary and maybe change its password.

Using cakePHP OpenId plugin on Windows

After all the praises I've read about OpenId, I decided to implement it in this CMS. My goal was to set an easy way for readers to leave comments without the need for registering to anything.

I thought it would also be great to add as a secondary, and easiest, mechanism for logging in.

So I downloaded and installed the openID component by cakebaker. As I'm running my dev environment under Windows, I had to set some settings.

Fixing the pluginPath

I've save the OpenId component in a plugin, and it has a clever mechanism to import the PHP OpenId library based on the folder it is saved.

Unfortunatly, the regexp used to know the name of the current plugin was throwing errors on Windows, due to the backslashes used in the file path.

I updated the getPluginName() method to this new one and it did the trick :

private function getPluginName() {
  $result = array();
  $ds = (Folder::isWindowsPath(__FILE__)) ? '\\\\' : DS;
  if (preg_match('#'.$ds.'plugins'.$ds.'(.*)'.$ds.'controllers#', __FILE__, $result)) {
      return $result[1];
  }

  return false;
  }

Basically it makes sure that the backslashes are correctly escaped under Windows.

Edit : I've sent this patch to the OpenId component author and it is now fixed in the latest versions.

Generating randomness

The second fix was to change the Auth_OpenID_RAND_SOURCE constant to null. This constant enable the library to generate randomness (AFAIK), by using the /dev/urandom.

This does not exists on Windows, so I added the following lines in my bootstrap.php

if (Folder::isWindowsPath(__FILE__)) {
    define('Auth_OpenID_RAND_SOURCE', null);
}

Connecting to SSL servers

The PHP bundle on Windows comes with cURLalready builtin, but without the bundle of the X.509 certificates of public CA. It means that the OpenId PHP library will refuse to connect to any CA using an SSL connection because it won't be able to check the certificate.

This does not happens on Linux, the list is correctly built in.

Fortunatly, we can pass a CURLOPT_CAINFO option to cURL to manually set a pre-defined bundle, and there already is one shipped with the PHP OpenId library.

All you have to do is add the following line on line 93 of the vendors/Auth/Yadis/ParanoidHTTPFetcher.php file :

curl_setopt($c, CURLOPT_CAINFO, str_replace('\\', '/', dirname(__FILE__)).'/../OpenID/ca-bundle.crt');

Paginated search results and custom url

I wanted for this blog a search feature, but I had some prerequisites for it :

• The search url could be bookmarked
• It should be paginated
• It should play well with my custom url starting with /blog

Defining custom urls

Here are the two routes I defined in my routes.php

Router::connect('/blog/search/:keyword',
    array('controller' => 'posts', 'action' => 'search'),
    array(
        'pass' => array('keyword'),
        'keyword' => '[^/]+'
    )
);
Router::connect('/blog/search/*', array('controller' => 'posts', 'action' => 'search'));

Going to/blog/search/*keyword* will start a search on the keyword, while going to /blog/search/would display a search form.

Writing the method

I started by creating a searchaction in my PostController, then creating a form submitting to this action, with a keywordinput field.

In the searchmethod, the first thing I do is checking if some POST data is submitted (coming from the search form). If so, I redirect to the same page, but passing the keywordas first parameter.

If no keywordis passed nor data submitted, I'll display a simple search form.

And finally if a keywordis specified, I'll do a paginated search on every posts whose nameor textcontains the keyword.

function search() {
  // We redirect to get it in GET mode
  if (!empty($this->data)) {
    return $this->redirect(array('keyword' => urlencode($this->data['Post']['keyword'])));
  }

  // Search index
  if (empty($keyword)) {
    return $this->render('search_index');
  }

  // Adding conditions to name and text
  $keyword = urldecode($keyword);
  $this->paginate = Set::merge(
    $this->paginate,
    array(
      'conditions' => array(
        'AND' => array(
          'OR' => array(
            'Post.name LIKE' => '%'.$keyword.'%',
            'Post.text LIKE' => '%'.$keyword.'%'
          )
        )
      )
    )
  );
  // Getting paginated result
  $itemList = $this->paginate();

  $this->set(array(
    'keyword' => $keyword,
    'itemList' => $itemList
  ));
}