This is obviously relevant for web applications and websites in general, which are available to the public. In such situations, the typical solution is to purchase an SSL certificate from a certification authority (such as RapidSSL or GoDaddy) and enable it within the webserver. This article is not about this.

What we’re going to talk about today is the security of Intranet web applications, such as a websites which are not publicly available, but somehow available to specific people having access to private networks.

Why is it important to secure intranet applications too?

Somebody might think that, since an application is not publicly available, it is not important to secure it with SSL. This is not true, and here’s why.

Think about a document storage web application that is running in the local area network (LAN) of your company. That application of course may contain many documents of different nature, such as customers’ and suppliers’ invoices and employees contracts and payment receipts.
While some of this documents might be visible to all the users of such web application, many others have to be visible to specific users only.
Let’s say that you implement a very secure ACL to ensure this security layer on your application and avoid unauthorized access to your sensitive data.
If you don’t serve your project using the HTTPS protocol, than all your documents are traveling in the cables of your LAN unencrypted, and anybody connected to the same network can easily scan your network and read those documents. Plus, a malicious user could use the same technique to sniff user passwords and easily access your secure web app with other peoples’ credentials (don’t you believe that? Check this tutorial).

That said, I hope you are all convinced now that SSL is important for every web application, even if it’s not public.

How to secure an intranet application

There are a few options to secure private web applications.

1. The dumbest option is to generate a self signed certificate. That does not solve the security problem though (read this to understand why).
2. Buy a standard certificate. This is perfectly acceptable but requires spending some amount of money every year.
3. Generate a certificate using Letsencrypt

If you haven’t heard about Letsencrypt before, go have a look at their website to learn more. In one sentence, it is an organization that aims at making the web more secure and gives you free SSL certificates for personal or professional use.
This is in my opinion the perfect solution for intranet applications (which tipically have budget constraints).

Generating SSL certificates with letsencrypt

Letsencrypt uses the ACME protocol to release SSL certificates in an automated manner.

That means you need an ACME client to get an SSL certificate. The official client is called certbot, it’s free and open source and it is easily installable in many UNIX-based operating systems. We’re not going to cover the installation of certbot, though, because it is out of scope. You can find easy tutorials for your operating system on the officiale page.

There are a few different ways to get a new certificate using certbot (such as apache, nginx, webroot, and so on. Read them here) but all of them have a very important common step to be performed which is domain ownership verification. This step is required by the certification authority to ensure that you are the owner, or at least you are allowed to manage, the DNS records for the internet domain you are requiring the certificate for. Skipping this step would mean that anybody could request a certificate for google.com or paypal.com and that of course would be a giant security problem.

How does Letsencrypt check domain ownership?

There are basically 2 ways:

• HTTP challenge: an HTTP request is sent to the domain name you requested the domain for. The server must respond to a specific endpoint with a specific content in order for the domain ownership to be verified.
• DNS challenge: letsencrypt servers will search on the DNS records of the requested domain for a specific TXT record, with a specific value.

HTTP challenge is the most common and the easiest, because certbot (and other ACME clients) can do that for you automatically. Unluckily, it requires that the domain name resolves to a public IP address which is freely reachable on the internet. But, remember, our domain is in the intranet and so this approach simply would not work in our scenario. That means we need to choose the DNS challenge option.

Generate a certificate using the DNS challenge

To do that, we need to use the manual strategy of certbot, by running a command similar to the following:

certbot -d private.mydomain.com --manual --preferred-challenges dns certonly

Of course adjust private.mydomain.com with your domain.
This command starts an interactive wizard:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Obtaining a new certificate
Performing the following challenges: dns-01 challenge for private.mydomain.com
------------------------------------------------------------------
NOTE: The IP of this machine will be publicly logged as having requested this certificate. If you're running certbot in manual mode on a machine that is not your server, please ensure you're okay with that.
Are you OK with your IP being logged?
------------------------------------------------------------------
(Y)es/(N)o: Y

------------------------------------------------------------------
Please deploy a DNS TXT record under the name _acme-challenge.private.mydomain.com with the following value:

LngHg_3lhXDJ_m3ArGTgtalz50uVCjXW5-zFVCulK8I

Once this is deployed,
------------------------------------------------------------------
Press Enter to Continue

At this point, as clearly requested by the script, you need to create a new TXT record in your DNS administration panel for _acme-challenge.private.mydomain.com with value LngHg_3lhXDJ_m3ArGTgtalz50uVCjXW5-zFVCulK8I

Once done, check that the DNS record propagated successfully (you can use this website to do that) than press enter to continue executing the script.

If all went well, you should see a success message:

IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at /etc/letsencrypt/live/private.mydomain.com/fullchain.pem.
Your cert will expire on 2017-05-16. To obtain a new or tweaked version of this certificate in the future, simply run certbot again.
To non-interactively renew *all* of your certificates, run "certbot renew"
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le

Good, our certificates are ready! You can find everything you need under the folder: /etc/letsencrypt/live/mydomain.com/ (of course adjust the path to meet your domain name). In that folder you’ll find 4 files:

• cert.pem: the actual certificate
• chain.pem: the certificate of the certification authority
• fullchain.pem: basically the merge of the above 2 files
• privkey.pem: your private key (never share this file with anybody)

Now you only have to tell your webserver to use the HTTPS protocol and where to get the certificate files. Check the official documentation for Nginx and apache to get started.

Recap

In this article we discussed about the risks of having a plain HTTP web application hosted inside our company’s private network and the need of an SSL certificate to make traffic flow securely inside our ethernet wires.

We then learned how to generate a free SSL certificate using Letsencrypt even if our webservers are not public.

First of all, if you still don’t know anything about Zend Expressive check it out: http://zendframework.github.io/zend-expressive/

Prerequisites:

• A decent grasp of PHP
• REST fundamentals
• Basic Zend Expressive knowledge

“Token based authentication works by ensuring that each request to a server is accompanied by a signed token which the server verifies for authenticity and only then responds to the request.”

So, we assume that we have a login page that will send the user credentials to our API endpoint (that will be http://api.mvlabs.it/api/v1/user/login in our example) and, if the given credentials are correct, then the API will respond with a token to the Front-End and the Front-End application will use the token to access the restricted resources in the following requests.

OK, so …how can I do that with Zend Expressive?

In Zend Expressive we could use a Middleware plugged into our pipeline to verify the token validity. So, for this scope we’ll use the tuupola/slim-jwt-auth middleware. It was originally developed for Slim but can be used with any framework that complies with the PSR-7 standard and of course, Zend Expressive is one of them! Yep, code reuse is a great thing enabled by Middleware! Ok, let’s take a look. First of all, we need to add to our Zend Expressive skeleton application the jwt auth middleware:

$ composer require tuupola/slim-jwt-auth

Now we need to add some configurations parameters that will be used by the JwtAuthentication Middleware. In fact, the only mandatory parameter is “secret” which is used to verify the token signature, only the issuer application has to know it and should never be disclosed in any way. So:

config/autoload/jwt-authentication.global.php

return [
   'jwt-authentication' => [
       "secret" => getenv("JWT_SECRET"),
       "path" => ["/"],
       "passthrough" => ["/api/v1/user/login", "/api/v1/ping"],
       "error" => function ($request, $response, &$arguments) {
           $data["status"] = "error";
           $data["message"] = $arguments["message"];

           return $response
               ->withHeader("Content-Type", "application/json")
               ->write(json_encode($data, JSON_UNESCAPED_SLASHES | JSON_PRETTY_PRINT));
       }
   ]
];

Some notes about the configuration above:

• The optional path parameter allows us to specify the protected part of your website.
• With optional passthrough parameter we can make exceptions to path parameter.
• error is called when authentication fails. It receives last error message as argument.

Now we can create our JwtAuthentication Middleware through a factory with our configuration defined above.

src/App/Factory/JwtAuthenticationFactory.php

namespace App\Factory;

use Interop\Container\ContainerInterface;
use Slim\Middleware\JwtAuthentication;

class JwtAuthenticationFactory
{
   public function __invoke(ContainerInterface $container)
   {
       $config = $container->get('config');
       return new JwtAuthentication($config['jwt-authentication']);
   }
}

Now is time to add the Jwt Authentication Middleware to the ZE middleware pipeline, then it will have the responsibility to determine if a token is valid for the protected endpoints.

config/autoload/middleware-pipeline.global.php

return [
   'dependencies' => [
       'factories' => [
	...
           Slim\Middleware\JwtAuthentication::class => App\Factory\JwtAuthenticationFactory::class,
	...
       ],
   ],
   'middleware_pipeline' => [
       'always' => [
           'middleware' => [
	       ...
               Slim\Middleware\JwtAuthentication::class,
               ...
           ],
           'priority' => 10000,
    ],
    'routing' => [
       'middleware' => [
            ...
       ],
       'priority' => 1,
    ],
       ...
   ],
];

At this point we need to define some routes in our configuration file, so let’s add the following:

• /api/v1/ping unprotected example endpoint
• /api/v1/user/login (unprotected) used for authentication and token generation
• /api/v1/user/:id the protected endpoint example that returns user specific data

config/autoload/routes.global.php

return [
   'dependencies' => [
       'factories' => [
           App\Action\AuthAction::class => App\Action\AuthFactory::class,
       ],
   ],

   'routes' => [
       [
           'name' => 'api.ping',
           'path' => '/api/v1/ping',
           'middleware' => App\Action\PingAction::class,
           'allowed_methods' => ['GET'],
       ],
       [
           'name' => 'api.auth',
           'path' => '/api/v1/user/login',
           'middleware' => App\Action\AuthAction::class,
           'allowed_methods' => ['POST'],
       ],
       [
           'name' => 'api.user.get',
           'path' => '/api/v1/user/:id',
           'middleware' => App\Action\UsersAction::class,
           'allowed_methods' => ['GET'],
       ],
   ],
];

Now, we need to add the logic that will handles the data sent by the login form, validate it against a database, and after determining that the credentials are valid, will generate the token. So, we’ll define the AuthAction Middleware that will have those responsibilities:

namespace App\Action;
namespace App\Action;

use App\Entity\User;
use App\Service\UserService;
use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\ServerRequestInterface;
use Zend\Diactoros\Response\JsonResponse;

class AuthAction
{
    private $userService;

    public function __construct(UserService $userService)
    {
        $this->userService = $userService;
    }

    public function __invoke(ServerRequestInterface $request, ResponseInterface $response, callable $next = null)
    {
        $post = $request->getParsedBody();
        $username = $post['username'];
        $password = $post['password'];

        $user = $this->userService->getUserByEmailPassword($username, $password);
        if ($user instanceof User) {
            $jwt = $this->userService->getAuthTokenForUser($user);

            return new JsonResponse([
                'time' => time(),
                'jwt' => $jwt,
            ]);
        }

        return new JsonResponse(['status' => 'error'], 401);
    }
}

Let’s take a detailed look to the token generation in the UserService:

namespace App\Service;

use App\Entity\User;
use Firebase\JWT\JWT;

class UserService
{

    ...

    public function geAuthTokenForUser(User $user)
    {
        $now = new \DateTime();
        $future = new \DateTime("now +2 hours");

        $random_number = openssl_random_pseudo_bytes(32); //generate using openssl_random_pseudo_bytes
        $jti = hash('sha256', $user->getId() . $random_number);
        $serverName =  $this->config['api']['endpoint'];
        /*
         * Create the token as an array
         */
        $data = [
            "iat" => $now->getTimeStamp(),  // Issued at: time when the token was generated
            'jti' => $jti, // Json Token Id: an unique identifier for the token
            'iss' => $serverName, // Issuer
            'nbf' => $now->getTimeStamp()+10, // Not before
            "exp" => $future->getTimeStamp(), // Expire
            'data' => [ // Data related to the signer user
                'userId' => $user->getId(), // userid from the users table
                'username' => $user->getUsername(), // User name
            ]
        ];

        /*
         * Encode the array to a JWT string. Second parameter is the key to encode the token.             *
         * The output token $jwt can be validated at http://jwt.io/
         */
        $jwt = JWT::encode(
            $data,
            $this->config['jwt-authentication']['secret'],
            "HS512"
        );

        return $jwt;
    }
}

In order to check that everything is OK, let’s do a test call to our authentication endpoint with postman, passing our authentication credentials.

Good, everything looks fine, so, at this point the client will have the token, and can store it using a storage mechanism like HTML5 Web Storage.

Now let’s retrieve a resource that is protected by our JWT mechanism. But first we need to set the Authorization header with the contents of the JWT in the format of Bearer [JWT]

As we can see in the image above, we can access to the restricted resource once that we passed our authentication token.

Summary

This is a very reliable method to secure your information. From here on, you can try to implement a token based authentication using ZE in your next API. Feel free to share your constructive feedback or ask a question. Drop a note below, we’ll be happy to receive your comments!

PSR-7

It all started with PSR-7, a set of common interfaces for representing HTTP messages in PHP, allowing application abstraction from the HTTP layer.

The most important aspect of PSR-7 is that it is framework independent and hence its interfaces can easily be used with multiple frameworks such as Symfony, Zend Framework, Laravel or any other framework.

This capability allows developers to write framework agnostic code with a wider opportunity to be reused in other projects, relying on the PSR-7 interfaces only.

PHP AND MIDDLEWARE

The other important key-word is middleware: once the PHP ecosystem gained a nice abstraction over HTTP messages, it was time to think about the best way to write web applications on top of this abstraction.

Citing Matthew Weier O’Phinney, the concept of middleware can be summarized in a single method signature:

function (request, response) : response

The idea is that objects representing the HTTP request and the HTTP response are passed to a callable, which does something with them to return an HTTP response.

This turns out to be a very easy and composable pattern, which had actually already been used with great success in several other programming languages.

The standard way to compose such middleware is to pass a third argument to the callable, so to allow for another layer of middleware to be invoked:

function (request, response, next) : response
{
    // do something on request

    response = next(request, response)

    // do something on response

    return response
}

Several frameworks, as Zend Expressive, Slim, Radar or Spiral rely nowadays upon the concept of middleware to create web applications.

MIDDLEWARE REUSE

Now that we have a highly composable pattern based upon a framework agnostic HTTP abstraction, we can actually start writing highly reusable components that would work on any PHP project. This definitely helps developers easing their daily job.

We can have middleware taking  care of logging all received requests, middleware to perform routing, middleware to manage authentication and some other to take care of user sessions. We use our framework to simply glue all this pieces together in the appropriate way.

The key question at this point should be: has anybody else already implemented the piece of middleware I need for the application I’m developing?

PHP-MIDDLEWORLD IS HERE!

Until now the answer to the previous question could only be gathered through Packagist, Github, or – in the worst case – google.

Having received so much from the community over the years, we thought it was about time to give something back. That’s how php-middleworld.com was born. With the aim to provide fellow PHP developers a means to quickly locate the middleware they need (often with multiple implementation choices).

THE FUTURE

We already have in mind several improvements for php-middleworld.com.

First of all we would like to enable fellow developers to add their own middleware, so that our database could grow with a shared effort.

Then we would also like to add PSR-15 compatible middleware support also, so to be ready when that proposal will be approved.

No need to get any deeper here, to find out more just head to php-middleworld.com and follow @phpmwd on Twitter!

Stay tuned!

The problem

Let’s say you are also developing a PHP application and you want to use docker in your development machine. You will notice that – if you use the official PHP images from the docker hub – sending e-mail out of the box simply won’t work. Lets’ see what I mean in practice, considering this simple index.php file, sending out an e-mail:

$result = mail("info@example.com", "Subject", "The very important email body");

if ($result) {
    echo "Email sent correctly";
} else {
    echo "Error sending the email";
}

Please note: using the mail command to send an email is the simplest way, but there are many more powerful libraries out there that could make things easier. We have chosen to use the mail command just to keep the example as simple as possible.

If we run it using docker with the following command:

docker run -p "80:80" -v `pwd`:/var/www/html --rm php:7-apache

and visit http://localhost with our browser, we’ll see that the mail command will fail, and we’ll get an “Error sending the email” message.

The reason why this happens is that the “mail” command relies on the “sendmail” system command inside the container, which is missing from the official PHP docker image.

While you could simply create a new image extending the original one and adding the sendmail command, I suggest using a different strategy.

Here at MV Labs we do use a tool called Mailhog (https://github.com/mailhog/MailHog) which acts as an SMTP server listening on port 1025 but instead of sending real emails to your recipients, it simply displays them in a handy web interface (which is listening, by default, on port 8025). This way you avoid the risk of sending your testing e-mails to the customer (it happened at least once to everybody) and you don’t have to deal with spam filters or other annoyances in the e-mail chain.

So, how can we achieve this? To make things easier, we can take advantage of docker-compose. Let’s start by adding the mailhog container to our services:

version: '2'
services:
  php:
    image: php:7-apache
    volumes:
      - .:/var/www/html/
    ports:
      - 80:80
    networks:
      - base
  mailhog:
    image: mailhog/mailhog
    ports:
      - 8025:8025
    networks:
      - base
networks:
  base:

Now, if we run docker-compose up and head our browsers to http://localhost:8025 we should see the mailhog simple but powerful web interface.

We’re not done yet, though, as we still need to tell the mail command to send emails through our new mailhog smtp server. Unluckily this requires the building of a custom docker image, since PHP doesn’t allow using the native mail command to connect to a SMTP server. For this reason, we need a tool called ssmtp (https://linux.die.net/man/8/ssmtp) which is basically a configurable sendmail command replacement, and which forwards the message to a SMTP server configured in its config file. We’ll need the following Dockerfile:

FROM php:7-apache

# install ssmtp
RUN apt-get update; apt-get install ssmtp -y

# tell php to use ssmtp's sendmail for email sending
RUN echo "sendmail_path=/usr/sbin/sendmail -t -i" >/usr/local/etc/php/conf.d/sendmail.ini

# tell ssmtp to use mailhog as the mail transport
RUN echo "Mailhub=mailhog:1025" > /etc/ssmtp/ssmtp.conf
RUN echo "FromLineOverride=Yes" >> /etc/ssmtp/ssmtp.conf

Then we’ll need to tell docker-compose.yml to build the image from the Dockerfile for our PHP container:

version: '2'
services:
  php:
    build: .
    volumes:
      - .:/var/www/html/
    ports:
      - 80:80
    networks:
      - base
  mailhog:
    image: mailhog/mailhog
    ports:
      - 8025:8025
    networks:
      - base
networks:
  base:

and build our containers again with:

docker-compose up -d --build

Now we’re ready to send our first “virtual” email. Just head your browser to http://localhost and you’ll get the message “Email sent correctly”. If you check your mailhog at http://localhost:8025 you’ll see your email!

Summary

Using a simple example, we have demonstrated one possible solution for sending test e-mails out of our web applications under development in a secure and handy way, by using some nice open source tools such as SSmtp and Mailhog inside a docker dev environment.

Let us know what you think in the comments area below and happy developing!

For starters, what is routing?

The fundamental principle of routing is to match incoming requests reaching an application and decide which fragment of code to execute. That could mean matching either an http request or a console requests (if you are interacting with a Command Line Interface application), and extracting the parameters that determine which snippet of code (IE which controller and then which action) to execute. The other basic principle is to assemble URLs and commands back. That is, from a set of parameters, to build the path of the route. In terms of an example, this means coming up with the /products/view/some_product URL starting from the controller = products, action = view, slug = some_product parameters*.

*I am aware this might not be the best possible example URL, but I hope it gets the message across in simple terms.

Why should we care about routing?

There are different reasons why routing is important. The simplest and most important one is that without routing your application won’t work Even in a working application, though, properly handling routing is important. That’s for different reasons, which can be summarized as follows:

- good URLs are easier to remember, type and share through word of mouth
- they also make it easier for search engines to properly index websites and web applications
- they allow for clean RESTful interfaces to be build
- they improve web application security, because they make it easier for parameters to be properly handled

Routing within the Zend Framework 2

Under the Hood

The basic unit of routing is a so called route, which is nothing but a PHP class implementing the following interface:

namespace Zend\Mvc\Router;

use zend\Stdlib\RequestInterface as Request;

interface RouteInterface
{
  public static function factory(array $options = array());
  public function match(Request $request);
  public function assemble(array $params = array(), array $options = array());
}

The RouteInterface includes the  static method factory, which is used to create new routes. Every route is constructed with arguments (usually fetched from specific project/module configuration file). The match method, in turn, accepts a new Request, and determines if the request matches current route. If true, it then returns a RouteMatch object, which is a container object, that includes the route name and most likely all route parameters and values. Last, we also have the assemble method, which takes options and parameters and builds the path of a route (usually an URL).

We will rarely need to worry about implementation details of these methods, though, because the framework will take care of passing the configuration and creating our routes according to our definitions in each module configuration directives (as we’ll shortly see).

Defining new routes

Routes for each module shall be defined within the module configuration file, which can be found in every module’s root folder. The default Zend Framework Skeleton app also has its own Module.php file in /config/modules.config.php:

The section dealing with routing is the one which goes under the router key; routes contains a list of our routes in array form:

return array(
  'router' => array(
    'routes' => array(
      'contact' => array(
        'type' => 'Literal',
        'options' => array(
          'route'    => '/contact',
          'defaults' => array(
            'controller' => 'Application\Controller\Contact',
            'action'     => 'index',
          ),
        ),
      ),
    ),
  ),
),

In the basic example above, we defined a route named “contact”, of type Literal (all different Route types will be covered in a forthcoming post). Basically we just told the framework to dispatch users requesting the ‘/contact’ URL to the Contact controller, and to have the IndexAction method to be executed.

What about route stacks

The ZF2 router is a collection of routes, based on a priority stack. As it was happening in Zend Framework 1, routes can be added to a route list; the last added route is the one tested first. The same now happens in ZF2; we can now also take advantage of the new priority attribute, though. This means that every route will have a priority value, and highest priorty route will be tested first. I’m sure this will make it easier to keep things organized without needing to worry much about route position within configuration files. This will also make it easier for custom configurations to be used to override third party default configurations.

Every router (there are different types, which we’ll take a look at shortly) implements this interface:

namespace Zend\Mvc\Router;

interface RouteStackInterface extends RouteInterface
{
  public function addRoute($name, $route, $priority = null);
  public function addRoutes(array $routes);
  public function removeRoute($name);
  public function setRoutes(array $routes);
}

All methods are self-explanatory, so I won’t go in further detail here. This is with the exception of the setRoutes method, which probably deserves some explanation. This method in fact replaces current routes in the stack with the new ones provided as an argument. So, it first removes current routes, then adding the new ones.

Router types

There are different implementations of the aforementioned Router base class:

• SimpleRouteStack: the most basic implementation, which we can use and extend to make our own router stack class.
• Http\TreeRouteStack: the most common kind of router, it is used to create a tree of routes to deal with http requests, using a B-tree algorithm to match routes (this ensures that evaluation is efficiently made, as we’ll shortly see).
• Console\SimpleRouteStack: used in the realm of CLI applications and use to take care of command line arguments. This kind of router will be processed when our application is run from a console terminal window.

Tree Routing

Zend Framework 2 introduces the Tree Routing concept as a big innovation. In fact, it’s very easy to use and its performance is great also: it’s much faster than its ZF1 counterpart. This is for multiple reasons: first off is the fact that the framework doesn’t try to match incoming requests with every single route anymore. This happens because every route can now have an unlimited number of children routes, which aren’t evaluated if parent route didn’t match the request first. Children routes are in fact only instantiated when the router matches or assembles their respective parent routes.

A Tree Route will consist of the following configuration:

• A base route, known as root route, which describes the base path (first part of an URL, for example);
• The may_terminate key, which is set to false by default, telling the router whether the route may terminate or not.
• The child_routes key, an array containing additional routes that stem from the base “route”. Each child route can itself be a Tree Route

Example:

return array(
  'router' => array(
    'routes' => array(
      'zf' => array(
      'type' => 'Literal',
      'options' => array(
        'route' => '/zf',
        'defaults' => array(
        '__NAMESPACE__' => 'Application\Controller',
        'controller'    => 'Article',
        'action'        => 'index',
      ),
    ),
    'may_terminate' => true,
    'child_routes' => array(
    'default' => array(
      'type'    => 'Segment',
      'options' => array(
        'route'    => '/:action',
        'constraints' => array(
          'action'     => '[a-zA-Z][a-zA-Z0-9_-]*',
        ),
      ),
    ),
  ),

Summary

In this first post about the Zend Framework 2 routing, I’ve introduced what’s a route, what’s routing, how the routing system works in the Zend Framework 2, and how it differs (especially from a performance point of view) from the ZF1 implementation. In my next post, I will be covering different HTTP route types we have at hand. Is there anything else you’d like to know? Just let me know with a comment below.

The problem: where assets should be placed?

The first problem to solve is: where my module’s assets should be placed to let me (the developer) forget about them? Because I should not care about assets provisioning, I would spend my time working on module itself! To simplify the problem, let me break it into two (not so) different use cases.

Shared assets

A module probably has many shared assets with other modules: javascript libraries, images, css files, etc. A common practice is to place all of them inside the project /public folder. Despite this is the simplest solution, it’s not the best. Copy/paste manually is never a good practice, besides copying all files inside the same folder will contribute to lose the explicit dependencies the module has with its assets.

Single module’s assets

In many cases a module has also some assets being used only by itself. We know that a module should focus on a single project functionality, so it should contain all the things that functionality needs to work well. The simplest answer now is probably the best: we should place all assets inside the module.

Some naive solutions

How should we make the web server aware that out modules have some assets that need to be visible inside our view scripts?

We could copy all assets (also those needed by a single module) inside public folder, every time one of them is added or updated. This is a bad practice, trust me!

We could use symlinks to let files be viewed inside another folder. Or we could do some apache htaccess tuning to let web server search files through all module folders. Luckily, there is a better way to handle things…

Welcome AssetManager module

AssetManager is a ZF2 module (https://github.com/RWOverdijk/AssetManager) aimed at managing assets, that resolve exactly our problem. It’s based on the Assetic asset management framework for PHP (https://github.com/kriswallsmith/assetic).

I don’t cover module provisioning and installation phases here because online there are straightforward and well organized articles about this (take a look on module’s wiki: https://github.com/RWOverdijk/AssetManager/wiki).

Instead, here I would like to explain how this module can be a real time (and headache) saver for all ZF2 developers.

After completing the install process, the module can take care of your assets through two simple steps:

1. Create a folder where assets should be placed. I typically suggest to use the /assets folder inside your module. Obviously you’re free to use whatever name you prefer. Sometimes you find assets inside a folder named /public, but this could be confused with the folder where the web server points to. So I suggest to choose a different name to make things clear. Use the folder you prefer, but keep in mind the rule of thumb: use always the same location inside all your modules, to make them more clear and readable.
2. Add the following rule to your module’s configuration file (/config/module.config.php – “assets” refers to the folder where your assets are placed):

'asset_manager' => array(
    'resolver_configs' => array(
        'paths' => array(
            __DIR__ . '/../assets',
        ),
    ),
),

With this two simple steps we are telling the AssetManager where the assets of our module are placed.

That’s it! Now when an asset is not found inside project /public folder, the AssetManager module’ll look inside our module /assets folder to find the requested file. From now you could link assets inside your views simply as follows:

<?php  echo $this->headLink()->prependStylesheet($this->basePath() . 
                                                '/css/aCssFile.css'); ?>

In this way, we don’t need to worry about where our css is physically located, just link it as if it was placed inside /public folder. So simple and so amazing! In this way you can place all module assets (both shared and exclusive) inside the module. So that module is now really reusable inside other projects, without hidden dependencies.

Pay only attention to avoid colliding file rules. It’s not uncommon to have modules containing files with the same name. In this case prefixing the asset path with the module’s name (or slug) could solve the issue (this can be done through the ‘map’ key inside configuration file; take a look on the code in the following paragraphs for an example).

This is the simplest use of AssetManager we could make, but this module has many more features that are worth an extra look. Let me define some terms the module uses (each of this maps to a key of module’s config file):

• Resolvers (resolver_configs key) allow to define where assets should be searched and how they are named / arranged. Many resolver types are available: MapResolver, CollectionResolver, PrioritizedPathsResolver and PathStackResolver (detailed infos and examples available here: https://github.com/RWOverdijk/AssetManager/wiki/Resolvers)
• Filters (filters key) allow make some processing on assets before serving them (eg. Minify minification, compiling of css files)
• Cache (caching key), as the name suggest, allows to choose which caching policy use to speed up assets serving time

An example

It’s a common practice to merge css files in order to limit the number of http requests (improving page serving speed). The AssetModule could help us to merge files automatically. Supposing our module has two css files (main.css and custom1.css), we could merge them together simply updating module’s config file as follows:

'asset_manager' => array(
    'resolver_configs' => array(
        'map' => array(
            'css/main.css' => __DIR__ . '/../assets/css/main.css',
            'css/custom1.css' => __DIR__ . '/../assets/css/custom1.css',
        ),
        'collections' => array(
            'css/merge.css' => array(
                'css/main.css',
                'css/custom1.css',
            ),
        ),
    ),
),

And the resulting view script is:

<?php echo $this->headLink()->prependStylesheet($this->basePath() . 
                                                '/css/merge.css'); ?>

More code samples can be found on official module wiki (https://github.com/RWOverdijk/AssetManager/wiki), that’s really well written and full of useful examples.

Conclusion

There are other ZF2 modules that provide asset management features. I personally like AssetManager because it’s really effective and simple to use. However is not important which module you decide to use, the important thing is that you should stop right now to manage assets manually. It’s a real boring and time wasting task, especially in the long run.

So now you’ve no excuses, this module requires no more than a few minutes to be installed and configured. Go and give it a try! And… after having configured your assets, don’t forget to enable AssetModule’s cache. That’s really a must to offer decent website performance and hence a good user experience.

For those who prefer a talk-like explanation, the extract of my Zend Framework Day presentation about this topic can be found here.

Image credit: http://www.flickr.com/photos/tracyleephoto/8322509672

The Problem

Script wasn’t outputting nor logging anything, so in order to find out more about the problem, I thought about logging last script execution datetime myself. I wanted to find out whether the problem was related to cron execution, or rather to some issue within the script itself. So I added following two lines (on top to the script):

$s_content = 'Script last run: '. date("d/m/Y h:i:s" . "\n");
file_put_contents('/tmp/cert_import.log', $s_content);

And executed it:

php ./importa_certificati.php

All I got was a nasty error message:

File size limit exceeded

At first I thought the problem might’ve been related to the specific file name I had picked (IE /tmp/cert_import.log). Did it exist already and was it already too big? I was sure I had checked things out. And actually, file wasn’t there upon a second check. So what was PHP ranting about? Could have it been a permission issue? Nope, script was being executed by root and was executable. So, where did the problem lie? I started to search for the problem by reverting the script to its original form, commenting out my additions:

// $s_content = 'Script last run: '. date("d/m/Y h:i:s" . "\n");
// file_put_contents('/tmp/cert_import.log', $s_content);

Problem magically disappeared. Weird, I thought. “Let’s see what happens if I prepare my string, but don’t it write to the file, I said to myself. I’m sure the problem lies there. Let’s check”:

$s_content = 'Script last run: '. date("d/m/Y h:i:s" . "\n");
// file_put_contents('/tmp/cert_import.log', $s_content);

File size limit exceeded

Whaat?!? Problem still there, despite no attempt to write to a file is even made. Things start to look awkward.

The Cause

What I had just discovered was that problem didn’t have anything to do with the file I was trying to write, but was related instead to the PHP process itself. In fact, wondering what other file size limit might’ve been reached, I came to think about logging. So, I opened php.ini and checked things out. Logging was configured indeed, pointing at /usr/local/zend/var/log/php.log through following directives:

; http://php.net/log-errors
log_errors = On

; Set maximum length of log_errors. [...]
log_errors_max_len = 1024

; Log errors to specified file. PHPs default behavior is to leave this value
; empty.
error_log = "/usr/local/zend/var/log/php.log"

I changed my working directory to /usr/local/zend/var/log/ and checked things out with ls:

[root@XXX ~]# cd /usr/local/zend/var/log/
[root@XXX log]# ls -alh
-rw-rw---- 1 apache zend 2.0G Dec 27 17:30 php.log

That exact 2Gb size looked suspicious, to say the least. Especially because the php.ini log_errors_max_len directive above had a 1024 value, not a 2048 one. So, I moved the log file out of its way, and gzipped it (so to save some space, without losing anything).

mv php.log
gzip php.log

I then run the script again. All went well this time, with no error being reported. Problem was caused by log file size indeed. Most likely no one cared about rotating it, believing that log_errors_max_len would do the job. As it’s explained on stackoverflow though, despite its possibly ambiguous name, log_errors_max_len only deals with the size of a single log message, not with the size of the log file itself.

Newly introduced problem was solved. But I was still wondering what might’ve caused the problem from the lines I had added in first place. Luckily I now had logs to check.

The timezone not set and include underlying issues

I opened the new php.log file and found a few entries like the following:

[05-Dec-2012 23:00:04 UTC] PHP Warning:  strtotime(): It is not safe to rely on the system's timezone settings. 
You are *required* to use the date.timezone setting or the date_default_timezone_set() function. 
In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. 
We selected 'Europe/Berlin' for 'CET/1.0/no DST' instead in /[...]/importa_certificati.php on line 14

Easy problem to fix. And also found the reason why log file filled up so quickly. Such message was written for every run of another script where the date function was used.

Still, I hadn’t solved my problem yet. Script would run from console, but wouldn’t from cron. Again, logs proved to be my friends, with entries like the following:

[27-Dec-2012 23:09:02 UTC] PHP Warning:  include_once(../classi/dbconnect.php): failed to open stream: No such file or directory in /[...]/importa_certificati.php on line 3
[27-Dec-2012 23:09:02 UTC] PHP Warning:  include_once(): Failed opening '../classi/dbconnect.php' for inclusion (include_path='.:/[...]') in /[...]/importa_certificati.php on line 3
[27-Dec-2012 23:09:02 UTC] PHP Warning:  include_once(../classi/utils.php): failed to open stream: No such file or directory in /[...]/importa_certificati.php on line 4
[27-Dec-2012 23:09:02 UTC] PHP Warning:  include_once(): Failed opening '../classi/utils.php' for inclusion (include_path='.:[...]') in /[...]/importa_certificati.php on line 4
[27-Dec-2012 23:09:02 UTC] PHP Fatal error:  Class 'utils' not found in /[...]/importa_certificati.php on line 10

I had just found out that problem was due to the script having relative includes in the following form:

<!--?php

include_once('../classes/dbconnect.php');
include_once('../classes/utils.php');

$s_pathCerts = __DIR__ . '/../rsync/data';

</pre-->

So everything worked when the script was tested by customer from local directory:

php ./importa_certificati.php

But once it was invoked by cron, it stopped working, since path wasn’t quite the same.
And, because no error messages related to the issue were collected in php.log, no one ever discovered about the problem with the script, and blamed cron settings instead.
Changing lines above tothe following solved the problem:

<!--?php

include_once(__DIR__ . '/../classes/dbconnect.php');
include_once(__DIR__ . '/../classes/utils.php');

$s_pathCerts = __DIR__ . '/../rsync/data';

</pre-->

Things to take home from this experience

1. If php can not write to its log file it will stop script execution, without giving a clear explanation of what’s going on
2. The log_errors_max_len directive is a bit ambiguous. It deals with each log record, not with the whole file size
3. You always need to set your timezone, if you don’t want to have your logs filled with crap
4. When including stuff, it’s wise to rely on the php __DIR__ or dirname(__FILE__) constructs rather than on relative paths. You can’t be sure the scritpt will be invoked from where you expect.
5. When troubleshooting it’s important to think “outside” the box. In this case the problem was related to the script itself, as much as it was related to its environment.
6. If you do write to logs, you need to check’em out every once in a while. Having their content e-mailed to you periodically is also a good idea

Elephant picture courtesy of http://www.flickr.com/photos/laughingsquid/2218075860/

Many other companies have experienced the same benefits and it’s for this reason that user stories are becoming a standard way to collect software requirements. If you’re not familiar with them, you can learn more reading here or getting this book. For now let’s just think of them as simple sketches on Post It notes, in the following form:

The good about User Stories

User stories don’t mention software features, focusing on user interactions and user benefits instead. Since they need to fit on a Post It sheet, they need to be short. Therefore, rather than being a list of detailed requirements, they end up being simple “reminders” of what will need to be discussed later on in the project life-cycle. Maybe. Yes maybe, because after having developed several stories, you might realize that several others do not need to be implemented at all. They are scratched from the project road-map altogether, before requiring significant design and development effort.

Stories are developed and delivered incrementally, in order of business priority. At each iteration (development phase) a batch of stories is selected, analyzed and turned into working software. Value is delivered to customers at regular intervals, much earlier than with traditional development processes.

Are User Stories a Silver Bullet?

I’m sure the above sounds great. And indeed it is. But let me tell you a story: several years ago I got to work with some folks from an Italian company, which is famous for its adoption of agile methodologies. I’ve learned a lot from that experience. They quickly explained everyone involved in the project what user stories were, then assigned each of the developers and customer representatives (around 15 people) a set of post it sheets, and finally said: “ok everybody, let’s now start writing user stories together”. Everyone started to think about some stories and gave his contribution. By the end of the day, we had collected several dozen Post It sheets.

All good, right? Well, not really. Sure we had put together a lot of stories. Problem is that later on, after we had developed them, we found out that many provided little or no value towards the project goals. And I’m sure we had missed out several others which would have. Don’t get me wrong. I’m not saying that stories are not the way to go, and that you shouldn’t measure things in retrospective. But development has a cost. And if you can avoid developing the wrong things from principle, well, I think that’s even better.

A problem of perspective

I’m now confident the problem we faced is a perspective one. Let’s consider user stories in the form we have seen above again. By starting our statements in the form

we are focusing on ourselves. And, in this example, if I think about myself shopping for some gifts, I think about as many filtering options as I can. And so – I’m sure – will do many other geeks, who just love to evaluate all possible available options. The problem is that intended system users will probably be different. They won’t know or care about Pareto optimals or multidimensional optimization. They’d probably just pick the best looking item, for the peace of their mind. System users more often than not have a different background, different expectations, and most likely different needs from ours. Which we need to take into account, and which we did not in the situation I’ve mentioned above.

Introducing Personas

In order to avoid this kind of issues, we introduced personas (for a good quick introduction see this book) into our requirements gathering process. We wanted to make sure we delivered software which satisfied intended users needs, not just our customers or ourselves. So, we started to create stereotyped characters of our intended users (whose characteristics were gathered either through interviews, observation or both) and then writing stories in the following form:

Now imaging implementing, or just estimating this story. At this point, you might be wondering who the heck Josie Black is. And that’s a good question. Let’s describe her as follows:

 ”Josie’s a 28 years old busy lawyer from Los Angeles, California. She’s always in a hurry, obsessed by her job and tends to forget about her friends’ birthdays. Therefore she looks for presents on the Internet, always at the very last moment. She has a budget in mind and doesn’t care much about making the perfect gift. A good one will be enough. She just wants to get the purchase job done as quickly as possible, so to get back to work. She’s annoyed by systems who do make a lot of questions and offer too many options.”

Now, what if Jodie’s profile was more similar to the following:

 “Josie’s a 68 years old retired business woman, who now lives in a small town in New England. When she was living in New York city, she loved to go shopping in malls and spending a lot of time looking for the perfect present for her friends and family members. Nowadays she enjoys the countryside, but misses shopping a lot. Luckily there’s the Internet, and that’s where she still does spend a lot of time searching for the best gifts. Before making a purchase she literally compares dozens or even hundreds of items.”

Now try to read the same user story above again. Do you feel the same way? If you need to make an estimate on development effort, do you think it’s going to take the same amount of time? Do you still think the story above is the best you could write? From our experience, the answer to these questions is almost always no. And that’s a good thing already, since the use of personas helps us being more accurate when writing stories and making initial estimates. What’s even more important, though, is the fact that when you properly consider the people who’ll use the system, you can easily ditch those stories which do not make sense at all. The process is no different from the traditional one, and everyone in the team can suggest a story. The point is that suggestions do not start from our perspective, but rather from that of our personas. And we’ve found this to be a significant switch. After all stories are collected, moreover, we review them and ask ourselves: “would this guy/gal, or anyone else of these folks here find this story useful”? If the answer is no, we simple drop the story.

Conclusion

The majority of our customers come to us with a set of requirements that they expect us to transform into software. When they accept to follow the process introduced above, interesting things emerge. The most significant one is surely the 50% of features they were requesting being cut off.  With personas it’s immediate for eveyone to find out about features which would be of no use for anyone. This is a fundamental point, which much improved our process and efficiency. But that’s only part of the game. As JJ Garrett points out, there are 2 things you need to address for an application to be successfull: user needs and business goals. The technique described above proved to be much helpful for better understanding user needs. A forthcoming post on this blog will talk about other techniques we do use in order to make sure business goals are also met.

Photo Credits: flickr.com/photos/eschipul/4160817135,
flickr.com/photos/coalitionforicc/6521548779, flickr.com/photos/judybaxter/4272568538