In the past few years, I've transitioned from engineering into product management at Zend, and it's been a hugely rewarding experience to be able to toss ideas over the fence to my own engineering team, and have them do all the fiddly tricky bits of actually implementing them!

Besides packaging long-term support versions of PHP, we also are publishing a product called ZendHQ. This is a combination of a PHP extension, and an independent service that PHP instances communicate with to do things like monitoring and queue management.

It's this latter I want to talk about a bit here, as (a) I think it's a really excellent tool, and (b) in using it, I've found some interesting patterns for prepping it during deployment.

What does it do?

ZendHQ's JobQueue feature provides the ability to defer work, schedule it to process at a future date and time, and to schedule recurring work. Jobs themselves can be either command-line processes, or webhooks that JobQueue will call when the job runs.

Why would you use this over, say, a custom queue runner managed by supervisord, or a tool like Beanstalk, or cronjobs?

There's a few reasons:

• Queue management and insight. Most of these tools do not provide any way to inspect what jobs are queued, running, or complete, or even if they failed. You can add those features, but they're not built in.
• If you are using monitoring tools with PHP... queue workers used with these tools generally cannot be monitored. If I run my jobs as web jobs, these can run within the same cluster and communicate to the same ZendHQ instance, giving me monitoring and code traces for free.
• Speaking of using web workers, this means I can also re-use technologies that are stable and provide worker management that I already know: php-fpm and mod_php. This is less to learn, and something I already have running.
• Retries. JobQueue allows you to configure the ability to retry a job, and how long to wait between retries. A lot of jobs, particularly if they rely on other web services, will have transient failures, and being able to retry can make them far more reliable.

So, what about queue warmup?

When using recurring jobs, you'll (a) want to ensure your queue is defined, and (b) define any recurring jobs at application deployment. You don't want to be checking on each and every request to see if the queues are present, or if the recurring jobs are present. Ideally, this should only happen on application deployment.

When deploying my applications, I generally have some startup scripts I fire off. Assuming that the PHP CLI is configured with the ZendHQ extension and can reach the ZendHQ instance, these scripts can (a) check for and create queues, and (b) check for and create recurring jobs.

As a quick example:

use ZendHQ\JobQueue\HTTPJob;
use ZendHQ\JobQueue\JobOptions;
use ZendHQ\JobQueue\JobQueue;
use ZendHQ\JobQueue\Queue;
use ZendHQ\JobQueue\QueueDefinition;
use ZendHQ\JobQueue\RecurringSchedule;

$jq = new ZendHQ\JobQueue();

// Lazily create the queue "mastodon"
$queue = $jq->hasQueue('mastodon')
    ? $jq->getQueue('mastodon')
    ? $jq->addQueue('mastodon', new QueueDefinition(
        QueueDefinition::PRIORITY_NORMAL,
        new JobOptions(
            JobOptions::PRIORITY_NORMAL,
            60, // timeout
            3, // allowed retries
            30, // retry wait time
            JobOptions::PERSIST_OUTPUT_ERROR,
            false, // validate SSL
    ));

// Look for jobs named "timeline"
$jobs = $queue->getJobsByName('timeline');
if (count($jobs) === 0) {
    // Job does not exist; create it
    $job = new HTTPJob('http://worker/mastodon/timeline', HTTPJob::HTTP_METHOD_POST);
    $job->setName('timeline');
    $job->addHeader('Content-Type', 'application/my-site-jq+json');
    $job->setRawBody(json_encode([
        'type' => MyApp\Mastodon\Timeline::class,
        'data' => [ /* ... */ ],
    ]);

    // Schedule to run every 15 minutes
    $queue->scheduleJob($job, new RecurringSchedule('* */15 * * * *'));
}

That's literally it.

The takeaway points:

• You can check for an existing queue and use it, and only define it if it's not there. You could also decide to suspend the queue and delete it before creating it, if you know that the existing jobs will not run with the current deployment.
• If you give a job a name (you don't actually have to, but it helps you identify related jobs far easier if you do), you can search for it. In the example above, if I find any jobs with that name, I know it's already setup, and I can skip the step of scheduling the job.

Running one-off jobs on deployment

Something else I also like to do is run one-off tasks at deployment. Often these are related to recurring tasks, and I might want to fetch the content at initialization rather than waiting for the schedule. In other cases, I might want to do things like reset caches.

Because these scripts run before deployment, which might mean restarting the web server, or, more often, waiting for the php-fpm container and/or web server to be healthy, I cannot run the jobs immediately, because there's nothing to answer them.

The answer to this is to queue a job in the future:

use DateTimeImmutable;
use ZendHQ\JobQueue\ScheduledTime;

$queue->scheduleJob($job, new ScheduledTime(new DateTimeImmutable('+1 minute')));

(I find it usually takes less than a minute for my FPM pool and/or web server to be online after running these scripts.)

The beauty of this approach is that my bootstrapping scripts now tend to be very fast, as I'm not trying to do all of this stuff before launching the site updates. The jobs then execute very soon after the site is up, and there's no noticeable differences in content or behavior.

Closing notes

I know I'm biased around ZendHQ. I'm also generally one of my own biggest critics. I had my team re-implement a lot of features that were present in Zend Server that I was never terribly keen on, and was hugely worried that we were going to make some of the same mistakes I felt we'd made with that product. However, the end result has been something that I am delighted to use, and which has opened up a ton of possibilities for how I build sites. The ability to warm my queues and manage them just like the rest of my PHP application is hugely powerful. I'm looking forward to seeing what others build with it!