Losing session on each request with cakePHP and Chrome

I finally found solution for one of the more tenacious bug I ever encountered. Share the joy !

I had a website working perfectly under Firefox but when browsing using Chrome, I noticed that my Session gets regenerated on each page load. Constantly. Creating hundred and hundred of useless session files.

And only with Chrome.

Since when using a browser should change the server behavior ? Well I don't exactly know what Chrome is doing with the referer but it seems that it is altering it in some ways.

And cakePHP forces the setting of session.referer_check to true, thus checking that multiple requests with the same PHPSESSID comes from the same url.

As one posted on php.net :

If you have a value specified for session.referercheck you may run into difficulty when someone accesses your site and attempts to log in with a mis- capitalized URL.  The logon will fail because any calls to sessionstart() will result in the existing session being trashed and a new one being created. This becomes a bigger problem when the logon is followed by a header("Location: ...") redirect, because the session_start() at the top of the page will fail.

Those two settings combined, and you got a hell of a mess. I first found a quick fix by forcing the setting of session_start() in app/webroot/index.php. But after more readingand debuggingI finally found the culprit.

Hacking your way through the fix

There is no easy way to prevent cake from setting this setting, but you can define your own session handler in the Session.save configure key.

Just create file named session_custom.php in app/config/ and set Configure::write('Session.save', 'session_custom'); in your core.php file.

And in that file, just drop the following lines (copy/paste from cake_session.php)

ini_set('session.referer_check', '');                    // Killing this f***ing config that was causing so much trouble with Chrome
ini_set('session.use_trans_sid', 0);                    // No session id in url
ini_set('session.name', Configure::read('Session.cookie'));    // Using custom cookie name instead of PHPSESSID
ini_set('session.cookie_lifetime', $this->cookieLifeTime);    // Cookie like time, depending on security level
ini_set('session.cookie_path', $this->path);                // Cookie path

Memcache keys not getting saved correctly

Update : We were initially blaming Memcache not correctly setting/getting keys. We should instead have blamed him for being over-zealous in clearing its list. See bottom of post for details.

Today we were having a bit of an issue with memcache. We had to write a thousand keys into it, reading them from a JSON file.

This code had been running happily on our server for the past two weeks. But as the number of keys increased, we came to spot that some keys seemed to be randomly cleared.

After further investigation (and a lot of hours), it appears that the keys were simply not set at all, or set to an empty/false value. This was really strange because it occured inside a loop on our JSON file, and never occurs for the same keys, nor everytime.

We took our server under high stress, forcing it to reload data very often, to trigger the bug once more. After a lot of test, we gather some more data, but nothing really came out. The missed keys were really random.

Finally, we decided to code a fast and dirty fix, until we found a better solution. Basically, we simply checks that the value is correctly saved after saving it, and if not, we retry. And we do so recursively.

function set($key, $val, $recursionLevel = 0) {
  Cache::write($key, $val);
  if (Cache::read($key)===false) {
    if ($recursionLevel>10) die('Possible infinite recursion);
    $this->set($key, $val, $recursionLevel);
  }
}

Note, that this will not work if one of your values is defined as false, otherwise you'll end up in an infinite loop.

Update

Well, the problem kept sporadically popping. And we finally found the REAL culprit.

Cake provides multiple Cacheconfig, using different settings. One would use a special config to set different cache duration. Each setting would use a different key prefix.

Unfortunatly, memcache has no way of finding keys starting with a special prefix. So when we do a Cache::clear(), it just wipes clear the whole memcache keys, no matter what config you specify.

We were running two different websites using the same memcache server, and when doing a clear on one website, it cleared keys on the second one as well.

We finally started one memcache server per site.

Multiple foreach() with references in php

It took me a couple of hours to debug : I had an array with values that shouldn't be here, values from another part of the application, and I couldn't figure out where that came from.

Well, it turns it was part of a documented feature of PHP :

Unless the array is referenced, foreach operates on a copy of the specified array and not the array itself. foreach has some side effects on the array pointer. Don't rely on the array pointer during or after the foreach without resetting it. Reference of a $value and the last array element remain even after the foreach loop. It is recommended to destroy it by unset().php.net

My code was as simple as :

$newArray = array();
$newArray2 = array();
foreach($list as $key => &$data) {
  $newArray[] = $data;
}
foreach($list2 as $key => &$data) {
  $newArray2[] = $data;
}

This resulted in the latest index of $newArray being set to one of $list2 values (didn't know exactly which). This is really counter intuitive.

Firebug 1.7 moving headers to raw POST source

Edit : The issue discussed here has been fixed in Firebug 1.9a2

It took me a whole day to track this bug down. At first I thought something was wrong in my request, I tried to replay it using Firebug and I kept hitting a "malformed request" error.

Debugging it in reverse order, using Wireshark and other tools I finally came to the conclusion that my request was indeed valid.

So what ?

The guilty part was indeed Firebug. When I ran a simple POST request from Flash, and inspect it from Firebug, I could spot that no Content-Length nor Content-Type headers were displayed in the "Headers" part. I then saw that the "source" part of the POSTtab contains my missing headers. What was that ?

HTTP POST Request are supposed to be split into two parts. First part are the headers, each separated by a \r\n. Then, after and empty line was the POSTraw content. Seeing that my headers gets mixed up with the POSTdata I thought that the request was malformed and a \r\n was added too early.

I checked with Wireshark and with PHP on the back end, and no, the request was indeed absolutly valid. Wireshark did not show any \r\n that shouldn't be there and PHP correctly parsed my request. The only issue was the details displayed by Firebug.

So it's a display bug ? I can live with it.

Well, actually, it's a little more than just a display bug. It kind of blocks my usual debugging workflow.

To debug an XHR request, I don't usually restart the whole page request just to debug one of the inner request.

The Firebug ability to "open request in new tab" is a time saver for me. I can play and replay the same request over and over until it's debuggued.

But with this bug, I can't use this feature. Opening the request in new tab will not send the slipped headers, and will instead send them in the POSTsource, resulting in a corrupted posted data.

I've posted an issue on the Firebug bug tracker. I hope it will get resolved soon. Until then, I'm using Live HTTP Headers to capture and replay the requests I need to debug.

Update

Seems like the bug come from Firefox internals and not Firebug. More details on this post.

Edit : Seems to be resolved in Firefox, will be available in next release (8.0 I guess).

Installing Ubuntu in VirtualBox to work as a development server

I was tired of trying to install Lighttpd on Windows (hint : don't try, it's a mess. I found no way to get vhosts and php as cgi at the same time). So I finally decided that Linux was the way to go.

I have always developped using Windows, and even if I'm slowly migrating to Ubuntu at home, I'm still using Windows 7 at work. The solution that best fits me was still using my Windows environment for coding, but using Ubuntu inside a VM to act as a server.

Let's see how I did that :

Required components

First, let's clarify what I'll have to install for this project : Lighttpd as the webserver, mysql and memcached for storing data and php5 to run cakePHP.

Step One : Running Ubuntu inside a VM

Start by downloading the Ubuntu iso (i's 687Mo, you'd better start this download first). I personnaly used the 10.04 Desktop version (I tried the 10.10 Server version first but I feel more confident with a UI).

Then, download the latest VirtualBox version(4.0.4 at the time of this writing).

And mount the iso in VirtualBox and start the install. This is pretty easy and you should be done in 10mn. Once the install is finished, you'll be prompted to reboot. Do not reboot just now. Instead, shutdown your guest machine (you can type sudo halt if the GUI does not work).

Once back in Windows, go to your VM config and unmount the install iso, otherwise the install procedure will be started again everytime you load your VM.

Once it's done, you can reboot the gues and once loaded, select the in VirtualBox Menu => Devices => Install guest additions. The next popup can take some long minutes to show, so be patient. This will install the guest additions that will allow you to easily communicate between the guest and host machines.

Once installed, reboot once more, and you should have the guest addition image sitting on your desktop.

Start a terminal and install the guest additions by doing :

cd /media/{name of your VBOX image}
./autorun.sh

This will finish the guest addition installation. Reboot your guest machine one last time and you'll be ready for step 2.

Step 2 : Installing the server stuff

We will now install the various components we'll need. You could probably install them all with one command but I'll split them in different line so you'll correctly see what's installed.

First, lighttpd :

sudo apt-get install lighttpd

Now, memcached

sudo apt-get install memcached

And mysql server. This one will prompt you to enter the root password you'll want.

sudo apt-get install mysql-server

Now we will install php5 as CGI (Lighty will run php as cgi), as well as the needed dependency so php can connect to both mysql and memcache.

sudo apt-get install php5-cgi php5-memcache php5-mysql

Ok, you should now have everything correctly installed. We will configure all that stuff later. You should now completely close your Virtual Machine and get back to Windows. It's time to configure your host and guest so they can correctly communicate

Step 3 : Enabling communication between host and guest

What we will do in this section is configure your network so the guest machine is considered part of your network and so you can connect to id using SSH and classic http.

First, go back to your VM config and change the Network mode to "Bridged". This will allow both machine to see each other in your network easily.

Then, define the Shared folder you want shared from your host in your guest. I shared my websites directories so Lighty could access them from the guest.

Now, start your VM, and type sudo ifconfig in a terminal. You should see your guest machine ip. Mine was 192.168.1.16.

Knowing that IP, you can define hosts in your Windows C:\Windows\System32\drivers\etc\hosts

You should also install openssh server so that you'll be able to connect to your guest machine using ssh :

sudo apt-get install openssh-server

We will now take care of the shared folder defined earlier. Selecting them in VirtualBox only made them available for mounting in the guest, but you'll now have to type some more commands to effectively see them. I needed to have access to a project I coded in my host (let's name it myproject).

Here's how I made it available to Lighty :

sudo mkdir /var/www/myproject
sudo mount -t vboxsf -o rw,uid=$(id -u),gid=$(id -g www-data) myproject /var/www/myproject

This will create a directory in /var/www to hold the project, and will then mount the shared folder to that directory. The uid/gid stuff indicate that you (the current user) is the owner and www-data is the group. Doing so will fix access rights issues you may have with Lighty.

You should have to re-run the last command on every login, so I strongly suggest you to put it in /etc/rc.local so it gets executed automatically.

Step 4 : Configuring all that stuff

I won't go into much details on how to configure lighty or php because it is not the scope of this post. I might post on that subject later, though. But here is one important step you shouldn't forget : uncomment the cgi.fix_pathinfo=1 line in your php.ini.

Don't forget to reload Lighty after changes to its config files by running:

sudo /etc/init.d/lightppd restart

I also had one additional side-effect on my install : installing php5 also installed Apache, causing Lighty to fail on startup because Apache was already using the port 80. I fixed it by removing every reference to Apache by running :

sudo update-rc.d -f apache2 remove

Final note :

If you followed this instruction, you should have a server running inside a VM that runs your website stored in your host. Isn't that pretty ?