Caching failed with panels in legacy mode
Did an deploy on one of the sites that we are working with, we did not get any errors on our staging server, but when we put it out on the production server we really had a big problem with caching. Suddenly cache would choke, display white screens of sadness, just caching part of the page, etc.
Crisis. At least. I tracked it down, with beads of sweat running down my face (the problem were on the production server for gods sake), and the problem were that Panels were in legacy mode and our custom layouts did not work. I found a couple of discussion on drupal.org on the subject. I updated the Skinr module to make panels running normal. Updated the database, cleared the cache a couple of times. And then, finally, everything worked ok again. I could wash my face, and make a phone call to the client to say that everything now was ok.
Lessons learned from this are several, but the most important was - never let panels work in legacy mode