Mike November

Not much going on at the moment outside of some client development work. A couple of projects that may throw up some interesting modules, but that will have to wait for a few weeks to see what's produced.

If you're in Sydney, there's a PHP User Group meetup that's showcasing SilverStripe. Andrew Short will be there to give a developer's insight to the platform, and Shane Weddell will be there to give a more business spin to the whole show.

Also, SilverStripe Australia is having a meetup to celebrate the end of year. There'll be a few things spoken about, but it will mostly be a case of a few beers and unwinding (at least, that's my plan!). I'll have a couple of module things to talk about, and we've got a partner giving an overview of migrating a site to SilverStripe from MySource Matrix. Come along!

When making sites that need a specific initial structure in place, I make use of requireDefaultRecords to create said structure. It makes things a lot easier for other developers working on the same project to have these things in automatically created and in place when they run dev/build after updating. While very handy for getting things up and running, there's a slight catch - requireDefaultRecords, by default, isn't run when you execute a unit test. This means that any functionality that you have that relies on these structures existing will likely break, with often confusing results.

How do you get around this? Make sure that the types you want requireDefaultRecords run for is specified!

    public function  setUpOnce() {
         parent::setUpOnce();
         $this->requireDefaultRecordsFrom[] = 'Page';
    }

There's been some discussion on the SilverStripe development list about some changes coming in SS3.0, one of which will be a frontend editing UI. Of course these plans are very much plans for the moment - the proper work of research, prototyping and putting together a clear set of features for such a UI has only just begun, which everyone needs to keep in mind. The more feedback and constructive criticism that goes into the process from those who develop with SS every day will mean that the end result will be better than if there was silence. Given that I've had an experimental frontend editing module floating around for about 12 months now, I figured I'd actually set up a demo site that people could play with to see my take on things, and what I'd like to see in future

See the demo here - Username: admin password: silverstripe. The db will reset itself once every hour or so, but please don't ruin it for everyone :).

Some things to know

• Click the "Edit Draft Page" button to edit the content after logging in
• You can edit the current page's contents, as well as any associated pages being displayed - when outputting a field in a template, it is wrapped in a call to create an editable field if possible.
• Styles can be defined in stylesheets that will appear in the Style... dropdown - these are prefixed with .wysiwyg- and automatically inserted.

Template modifications look like

    

        $EditableField(Content)

        
        
Child pages
        
Just to demonstrate editing other page contents contained in control lists 
        
        
Title
        
$EditableField(Title)
        
Content 
        
$EditableField(Content)
                
    

This is available via my github account, but is experimental code for now. This DOES NOT have anything to do with the SS3.0 plans at all - this is something that has been sitting around for a while but may prove helpful in getting people thinking about what a proper solution should have (ie, everything that this doesn't! :)).

After much tweaking and stalling, the QueuedJobs module has been released (and submitted to SilverStripe.org, should be up soon). What does it provide?

The Queued Jobs module provides a framework for SilverStripe developers to define long running processes that should be run as background tasks. This asynchronous processing allows users to continue using the system while long running tasks proceed when time permits. It also lets developers set these processes to be executed in the future.

In essence, the goal is to not leave users with a seemingly 'hanging' connection that may eventually time out when they trigger an action that might take a while to process. Not many actions in SilverStripe do this, but your site might have particular need for it. Some areas where we're using it

• Generating google sitemap.xml files for sites with hundreds and thousands of pages.
• Publishing large subtrees of pages
• Re-indexing content in systems that use external search engines such as Solr.

If you want to test it out, download it from GitHub or SilverStripe.org. After extracting it to the queuedjobs folder, and running dev/build, you'll need to make sure you have a cronjob setup to run the main processor (preferably as the webserver user).

*/1 * * * * php /path/to/silverstripe/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask */15 * * * * php /path/to/silverstripe/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=2

See the wiki page for more info.

I recommend you test out the GenerateGoogleSitemapJob to get a feel for what's going on under the covers (and it actually has a functional benefit!). To create the initial instance, go to http://path.to.silverstripe/dev/tasks/CreateDummyJob?name=GenerateGoogleSitemapJob which will create it (it will recreate itself as it processes. To make things easier, I'll step through the code so you get an idea of what's important when doing your own jobs

    public function __construct() {
        $this->pagesToProcess = DB::query('SELECT ID FROM "SiteTree_Live" WHERE "ShowInSearch"=1')->column();

        $this->currentStep = 0;

        $this->totalSteps = count($this->pagesToProcess);
    }

When constructing the job, I get a list of all the Live pages on the site (these are the only ones that are going to be indexed by google) that are set to show in the search. We're only interested in the ID of these pages though, not the actual objects. This is because we're going to store the full list of IDs of the pages we need to process - the $this->pagesToProcess variable here gets serialized and stored in the database between processing events, enabling us to stop and start processing at any time.

    public function getJobType() {
        if ($this->totalSteps > 100) {
            return QueuedJob::LARGE;
        }

        return QueuedJob::QUEUED;
    }

Here we're arbitrarily making the judgement that > 100 pages to generate an XML file for is enough for the job to be classified as 'large'. There's no real processing difference for this at the moment; the main reason for doing so is to not clog up one of the queues with a job that will take several minutes to execute.

    public function getSignature() {
        return md5(get_class($this));
    }

To prevent multiple instances of the same job being added to a queue, each job defines a signature. The base AbstractQueuedJob defines a default that should be good enough for 95% of jobs, but in some cases you want to ensure that a job is the only one of its kind, regardless of parameters.

    public function  setup() {
        parent::setup();
        increase_time_limit_to();
        
        $tmpfile = tempnam(getTempFolder(), 'sitemap');
        if (file_exists($tmpfile)) {
            $this->tempFile = $tmpfile;
        }
    }

The setup() method is called just before a job starts for the first time. In this case, we're wanting to make sure that a temporary file (that we're going to build the sitemap.xml file into first) exists for us to work with.

    public function prepareForRestart() {
        parent::prepareForRestart();
        // if the file we've been building is missing, lets fix it up
        if (!$this->tempFile || !file_exists($this->tempFile)) {
            $tmpfile = tempnam(getTempFolder(), 'sitemap');
            if (file_exists($tmpfile)) {
                $this->tempFile = $tmpfile;
            }
            $this->currentStep = 0;
            $this->pagesToProcess = DB::query('SELECT ID FROM SiteTree_Live WHERE ShowInSearch=1')->column();
        }
    }

The prepareForRestart() method is executed whenever the job has been paused then restarted. It could have been restarted by a user manually pausing, or an error that caused it to stop. Either way, it gives us a chance to check the state of the job, and if necessary restart it. We could just as easily flag the job as complete here and not continue, but in this case we're making sure our temporary file still exists, and if it doesn't, creating a new one from scratch.

    public function process() {
        $remainingChildren = $this->pagesToProcess;

        // if there's no more, we're done!
        if (!count($remainingChildren)) {
            $this->completeJob();
            $this->isComplete = true;
            return;
        }

        // lets process our first item - note that we take it off the list of things left to do
        $ID = array_shift($remainingChildren);

        // do some processing work that adds content to $tmpfile
        // ... snip ...

        // and now we store the new list of remaining children
        $this->pagesToProcess = $remainingChildren;
        $this->currentStep++;

        if (!count($remainingChildren)) {
            $this->completeJob();
            $this->isComplete = true;
            return;
        }
    }

The process() method is where all the actual work for this job happens, but it still needs to do a minimum of things to keep the container happy and in sync with things. First, it retrieves the list of pages still to be processed, and checks to see if there's anything left, if not marking the job complete. Next, it does the actual work with the next item in the list, then updates $this->pagesToProcess to make sure that next run through is onto the next item. It updates how many steps have been processed, then does another check to see whether the job has completed.

    protected function completeJob() {

        // ... snip ... 

        if (file_exists($this->tempFile)) {
            unlink($this->tempFile);
        }

        $nextgeneration = new GenerateGoogleSitemapJob();
        singleton('QueuedJobService')->queueJob($nextgeneration, date('Y-m-d H:i:s', time() + self::$regenerate_time));
    }

Finally, our completeJob() method actually copies our temp file to the right location, then cleans up the old file. Lastly, this job creates a NEW job and adds it to the queue to be processed at a date in the future; in this case it executes once every day.

Okay, that was a whole lot of words, but hopefully gives an idea of what's involved in writing a queued job, or more to the point, what you don't have to worry about. The framework around this manages everything to do with error handling and reporting, including automatically pausing and restarting jobs and notifying on broken jobs. It manages the persistence of job state so that jobs can be picked up after they've been paused and still continue on. It also manages the scheduling of jobs in the future, so you can use the module almost as a cron replacement.