The Problem

Script wasn’t outputting nor logging anything, so in order to find out more about the problem, I thought about logging last script execution datetime myself. I wanted to find out whether the problem was related to cron execution, or rather to some issue within the script itself. So I added following two lines (on top to the script):

$s_content = 'Script last run: '. date("d/m/Y h:i:s" . "\n");
file_put_contents('/tmp/cert_import.log', $s_content);

And executed it:

php ./importa_certificati.php

All I got was a nasty error message:

File size limit exceeded

At first I thought the problem might’ve been related to the specific file name I had picked (IE /tmp/cert_import.log). Did it exist already and was it already too big? I was sure I had checked things out. And actually, file wasn’t there upon a second check. So what was PHP ranting about? Could have it been a permission issue? Nope, script was being executed by root and was executable. So, where did the problem lie? I started to search for the problem by reverting the script to its original form, commenting out my additions:

// $s_content = 'Script last run: '. date("d/m/Y h:i:s" . "\n");
// file_put_contents('/tmp/cert_import.log', $s_content);

Problem magically disappeared. Weird, I thought. “Let’s see what happens if I prepare my string, but don’t it write to the file, I said to myself. I’m sure the problem lies there. Let’s check”:

$s_content = 'Script last run: '. date("d/m/Y h:i:s" . "\n");
// file_put_contents('/tmp/cert_import.log', $s_content);

File size limit exceeded

Whaat?!? Problem still there, despite no attempt to write to a file is even made. Things start to look awkward.

The Cause

What I had just discovered was that problem didn’t have anything to do with the file I was trying to write, but was related instead to the PHP process itself. In fact, wondering what other file size limit might’ve been reached, I came to think about logging. So, I opened php.ini and checked things out. Logging was configured indeed, pointing at /usr/local/zend/var/log/php.log through following directives:

; http://php.net/log-errors
log_errors = On

; Set maximum length of log_errors. [...]
log_errors_max_len = 1024

; Log errors to specified file. PHPs default behavior is to leave this value
; empty.
error_log = "/usr/local/zend/var/log/php.log"

I changed my working directory to /usr/local/zend/var/log/ and checked things out with ls:

[root@XXX ~]# cd /usr/local/zend/var/log/
[root@XXX log]# ls -alh
-rw-rw---- 1 apache zend 2.0G Dec 27 17:30 php.log

That exact 2Gb size looked suspicious, to say the least. Especially because the php.ini log_errors_max_len directive above had a 1024 value, not a 2048 one. So, I moved the log file out of its way, and gzipped it (so to save some space, without losing anything).

mv php.log
gzip php.log

I then run the script again. All went well this time, with no error being reported. Problem was caused by log file size indeed. Most likely no one cared about rotating it, believing that log_errors_max_len would do the job. As it’s explained on stackoverflow though, despite its possibly ambiguous name, log_errors_max_len only deals with the size of a single log message, not with the size of the log file itself.

Newly introduced problem was solved. But I was still wondering what might’ve caused the problem from the lines I had added in first place. Luckily I now had logs to check.

The timezone not set and include underlying issues

I opened the new php.log file and found a few entries like the following:

[05-Dec-2012 23:00:04 UTC] PHP Warning:  strtotime(): It is not safe to rely on the system's timezone settings. 
You are *required* to use the date.timezone setting or the date_default_timezone_set() function. 
In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. 
We selected 'Europe/Berlin' for 'CET/1.0/no DST' instead in /[...]/importa_certificati.php on line 14

Easy problem to fix. And also found the reason why log file filled up so quickly. Such message was written for every run of another script where the date function was used.

Still, I hadn’t solved my problem yet. Script would run from console, but wouldn’t from cron. Again, logs proved to be my friends, with entries like the following:

[27-Dec-2012 23:09:02 UTC] PHP Warning:  include_once(../classi/dbconnect.php): failed to open stream: No such file or directory in /[...]/importa_certificati.php on line 3
[27-Dec-2012 23:09:02 UTC] PHP Warning:  include_once(): Failed opening '../classi/dbconnect.php' for inclusion (include_path='.:/[...]') in /[...]/importa_certificati.php on line 3
[27-Dec-2012 23:09:02 UTC] PHP Warning:  include_once(../classi/utils.php): failed to open stream: No such file or directory in /[...]/importa_certificati.php on line 4
[27-Dec-2012 23:09:02 UTC] PHP Warning:  include_once(): Failed opening '../classi/utils.php' for inclusion (include_path='.:[...]') in /[...]/importa_certificati.php on line 4
[27-Dec-2012 23:09:02 UTC] PHP Fatal error:  Class 'utils' not found in /[...]/importa_certificati.php on line 10

I had just found out that problem was due to the script having relative includes in the following form:

<!--?php

include_once('../classes/dbconnect.php');
include_once('../classes/utils.php');

$s_pathCerts = __DIR__ . '/../rsync/data';

</pre-->

So everything worked when the script was tested by customer from local directory:

php ./importa_certificati.php

But once it was invoked by cron, it stopped working, since path wasn’t quite the same.
And, because no error messages related to the issue were collected in php.log, no one ever discovered about the problem with the script, and blamed cron settings instead.
Changing lines above tothe following solved the problem:

<!--?php

include_once(__DIR__ . '/../classes/dbconnect.php');
include_once(__DIR__ . '/../classes/utils.php');

$s_pathCerts = __DIR__ . '/../rsync/data';

</pre-->

Things to take home from this experience

1. If php can not write to its log file it will stop script execution, without giving a clear explanation of what’s going on
2. The log_errors_max_len directive is a bit ambiguous. It deals with each log record, not with the whole file size
3. You always need to set your timezone, if you don’t want to have your logs filled with crap
4. When including stuff, it’s wise to rely on the php __DIR__ or dirname(__FILE__) constructs rather than on relative paths. You can’t be sure the scritpt will be invoked from where you expect.
5. When troubleshooting it’s important to think “outside” the box. In this case the problem was related to the script itself, as much as it was related to its environment.
6. If you do write to logs, you need to check’em out every once in a while. Having their content e-mailed to you periodically is also a good idea

Elephant picture courtesy of http://www.flickr.com/photos/laughingsquid/2218075860/