Blanksy's Gulp Task Manager (BGTM)

Breaking Changes

As of version 2, there are a number of breaking changes:

Gulp 4 Support

Support for Gulp 4.x has broken support for verion 3.x, you must ensure you are using the latest gulp-cli as well.

Engine Changes may require updates

Engines need to be updated to include a 3rd parameter. This third parameter is used if you are using the streams for your engine to allow for correctly returning a complete state to gulp. See the 'Engine' section for further information.

Introduction

This is a wrapper for gulp that allows for a consistent template-based approach to project setup for gulp absed on your development teams standards for build processes.

Why?

This task manager is a solution to a problem that has been faced countless times in development when dealing with large amounts of projects and a constantly changing team of developers and it usually consists of 3 main challenges that are faced, these are:

Consistency & Team Best Practices

Challenge

You have a team, they have outlined a build process including tasks that they consider best practice, however these tasks are in their own files, still in gulpfile.js and they clone this from project to project.

What generally happens with this process is that you copy it from project to project, and you tweak it to optimise some steps by changing a build step here, a build step there and then all of a sudden you have all your projects all similar but all slightly different in subtle ways, they have been updated for the latest best practices but the changes were not ported back to the other projects.

This inconsistency can be an absolute pain for maintaining a certain level of quality across all your projects.

Solution

The concept that this framework goes with is to create engines to solve this problem.

What are engines? You can think of them as an NPM Package containing your best practices for a specific build pipeline.

Example: You currently have a build pipeline that does the following:

• Gets the source file(s)
• Concatenate the source files into a single file
• Minify the file to reduce the filesize.
• Generate the sourcemaps for that file.
• Ensure that the file always has an extension of .min.js
• Always have the sourcemap have the same slug as the main file
• Output the new minified file.

This pipeline can be considered to be a set of best practices you follow and are the rules that make up an engine that should apply to all your projects.

Maintainability of the Build Process Best Practices

So we have talked about the Consistency and Best Practices which brings us on to the Maintainability of the build process which constantly shifts as new concepts come out and new idea's are formed.

Challenge

Say that maintain a large number of projects or have projects being outsourced and you want to keep a consistent standard for your builds.

What can happen is a developer modifies the build steps in one project.

This then brings that project out of sync with your established best practices and will have the potential to cause chaos a little down the line when you go to integrate it with CI, pick the project up for maintenance expecting it to work one way but works another entirely different way.

But now you want to port those changes back, to do that you now have to go through each project manually and that could lead to issues in one project but not the other.

Solution

Again with the concept of engines on your side and the power of npm you have the ability to update the underlying best practices you have specified and bring those projects up to date.

For this you will need to install the following:

npm install -g npm-check-updates

And then run the following commands

ncu -u
npm install

It does take a little change of process of course as you need to think if what you are adding is going to be a part of your best practices, you need to then checkout or even create the engine code and then publish this new engine into NPM.

This may seem tiresome and a little long winded however you are actually setting a precedent by saying the rules that make up my best practices are important and should be considered before updating them.

CI (Continuous Integration) Friendly

Challenge

Generally with CI implementation you want to keep it repeatable, as repeatable makes it very easy to setup new projects as the steps for building a project are based on a template that does a specific set of steps and relies on things being done in a similar way.

Because the setup process is templated and repeatable you want to be able to ensure that the build files all have the same base level tasks as well to ensure that when you want to build a production ready version you can do so, when you want to trigger a build you can.

Solution

The following tasks are ALWAYS available:

• build
• compile (alias of build)

There is a --production and --environment argument that can be passed in for the purposes of only doing specific build steps during the build as part of the engine makeup.

So as a CI implementor you would implement the frontend build by running:

gulp build

or

gulp build --production

or

gulp build --environment <ENVIRONMENT_NAME>

Getting Started

To get started create your project using

cd /to/your/project/directory/
npm init
npm i --save-dev gulp bgtm

How to structure your gulpfile.js

var gulp = require('gulp');
var tm = require('bgtm')(gulp);

... Any task require statements ...

... Any tasks to be added to the manager ...

// ALWAYS run this method last.
tm.run();

Tasks

The task manager contains tasks, just like gulp. Except that the manager has rules surrounding how the tasks can be run.

The following tasks are ALWAYS defined automatically without you having to add them:

• compile
• build
• watch
• default

compile is an alias of build.

build is the default task that runs all the tasks needed to build a project, this is specified as part of the task addition.

watch sets the file watchers up to run a specific task.

default ALWAYS runs the build task then it runs the watcher to watch for file changes.

Adding a task

The below is an example only

First install the engine npm package:

npm i --save-dev bgtm-engine-scss

Then load the dependency:

var scss = require('bgtm-engine-scss');

Then add the engine to the task manager:

tm.add('taskName', {
    runOnBuild: false,
    runOnBuildEnvironment: 'all',
    watch: false,
    watchSource: [],
    liveReload: false,
    engine: scss,
    engineOptions: {
        src: ['static/css/src/main.scss'],
        dest: 'static/css/
    }
})

Option Outline

Option	Type	Value
runOnBuild	boolean	Adds this task to the list of tasks that should be run when gulp build is run.
runOnBuildEnvironment	string or array	The ability to only run this task when a specific environment is specified. You add the environment or environment names you want to be able to run this task for.
watch	boolean	Adds this task to the list of tasks that have files being watched and should be actioned when you update a certain set of files.
watchSource	array	An array of patterns for the files/file types to be matched when watching for file changes.
liveReload	boolean	Set this to true if you are using livereload to automatically update your browser when the task has finished updating the files.
engine	function	The callback (npm package or custom function) that can be used to run the gulp pipeline.
engineOptions	object	An object containing all options for the enging to function, such as source and destination folders etc.

Group tasks into a Sequence Group

You can also group tasks so that they can be run in the order that they appear by using the sequence task name.

There are some limitations:

• sequence groups cannot live reload and this is turned off for the groups
• the tasks in the group cannot be run on build, watched or live reload.

To use this functionality you can wrap them in the following construct:

task.addSequence(
    'static-files',
    {
        runOnBuild: true,
        watch: true,
        liveReload: true
    },
    function (tm) {

        // Normal tasks get added here.

    }
);

addSequence Options

Code

task.addSequence(taskName, sequenceOptions, callback);

Options

Option	Type	Value
taskName	string	The name to give the sequence so that it can be run independently.
sequenceOptions	object	The options to pass to the sequence on when to run.
callback	function	The callback used to wrap the tasks, it contains one parameter which is the instance of BGTM.

sequenceOptions

Option	Type	Value
runOnBuild	boolean	Adds this task to the list of tasks that should be run when gulp build is run.
runOnBuildEnvironment	string or array	The ability to only run this task when a specific environment is specified. You add the environment or environment names you want to be able to run this task for.
watch	boolean	Adds this task to the list of tasks that have files being watched and should be actioned when you update a certain set of files.
watchSource	array	An array of patterns for the files/file types to be matched when watching for file changes.

Running the tasks

You would run the tasks exactly the same way you would run a normal gulp process.

gulp taskName

Environment Mode

The task manager has an environment system that allows for customisation of actions and what can run on build based on the environment you specify during build.

There are 3 default environment reserved words.

ENVIRONMENT	JS ACCESS
all	tm.environment.ALL
development	tm.environment.DEVELOPMENT
production	tm.environment.PRODUCTION

You can also specify your own by simply implementing this in the:

• task option runOnBuildEnvironment value.
• hard-wiring it as part of your own custom engine.

How to change environment mode

If the environment argument is not specified then the default environment is development

You can change the environment mode that the gulp task runner is running under by using either:

--environment <ENVIRONMENT>

or

--production

The functionality of --environment production and --production are exactly the same and are interchangeable, so use whichever you prefer.

The reason for the difference is that the production flag has a special function in that it also sets the process environment for node to production mode as well.

How runOnBuildEnvironment words

This is an option that is available for every task that only adds the task to the global build task when that specific environment is run.

By default this is set to all which means that it runs no matter what environment is specified using the --environment <ENVIRONMENT> argument.

If you wish to only run a task in development mode then you can specify this value as being development.

This is very useful if you want a copy task to be only run during development but not during the CI processes for production / release etc.

e.g. The following task will only run when the development and release environments are specified during the gulp runner.

tm.add('taskName', {
    runOnBuild: true,
    runOnBuildEnvironment: [
        'development',
        'release'
    ],
    ...
})

To run this task you will need to use either:

gulp
gulp --environment development
gulp --environment release
gulp build
gulp build --environment development
gulp build --environment release

The following will not run this task

gulp build --production
gulp build --environment production
gulp build --environment anyotherenvironment

How to use the environments in an engine.

You can read more on building engines below, however the code effectively equates to the following:

if(tm.isEnvironment(tm.environment.PRODUCTION)) {
    // Perform the action you want for production environment.
}
if(tm.isEnvironment(tm.environment.DEVELOPMENT)) {
    // Perform the action you want for development environment.
}
if(tm.isEnvironment('release')) {
    // Perform the action you want for custom release environment.
}

Engines

Pre-Existing Engines

There are a number of engines already built that can be used if you find that they match your own best practices, however these may not suit your purposes.

• bgtm-engine-scss
• bgtm-engine-js
• bgtm-engine-images
• bgtm-engine-svgsprite

Creating an Engine

Engine Types

Stream Based

Stream based engines are just standard tasks created in the same way as normal with gulp, except in a package which can be reused without changing the rules behind it for consistency in the way it works.

Standalone

A standalone engine is an engine that does not need to use the stream but can be used independently without the need of the gulp stream.

Using your new engine

Create the index.js and package.json files for your project like a normal NPM package.

Ensure that any dependencies are added to your package.json file.

Example Engine Code

package.json

{
  "name": "bgtm-engine-example",
  "version": "0.0.1",
  "description": "",
  "author": "Blanksy",
  "license": "MIT",
  "dependencies": {},
  "main": "index.js",
  "devDependencies": {
    "gulp-plumber": "^1.1.0",
    "gulp-rename": "^1.2.2"
  }
}

index.js

Stream Based

var plumber = require('gulp-plumber');
var rename = require('gulp-rename');

module.exports = function(taskManager, engineOptions, done) {
    var task = this.src(engineOptions.src);
    task.pipe(plumber());
    if(tm.isEnvironment(tm.environment.PRODUCTION)) {
        // When in production mode rename the files.
        task.pipe(rename({extname: '.min.css'}));
    }
    task.pipe(this.dest(engineOptions.dest));
    task.pipe(done());
    return task;
};

Standalone

module.exports = function(taskManager, engineOptions, done) {
    // Sample Async Call
    setTimeout(function() {
        done();
    }, 10000);
};

If you need to access gulp this can be achieved by using this as part of the engine code.

There are also 3 parameters to be aware of that make up the engine itself, these are:

Parameter	Type	Value
taskManager	TaskManager	The instance curently running for BGTM itself.
engineOptions	object	The object containing all the options passed into the engine.
done	function	This is the callback that is called when your engine has finished its processing. If you are using a stream based engine use .pipe(done()) otherwise you can use done() when your engine has finished processing.

When you are happy with the engine, commit the code to git and then do an npm publish

Inline Callback (Not Recommended)

Whilst I do not recommend this approach as it goes against the spirit of this manager, it is important to outline that this can be done and how to do it.

In the engine option for the task addition you can (should not) specify a callback function as follows:

tm.add( 'taskName', {
    runOnBuild: true,
    watch: true,
    watchSource: [
        'static/fonts/**/*'
    ],
    liveReload: false,
    engine: function(taskManager, engineOptions, done) {
        var task = this.src(engineOptions.src)
        task.pipe(plumber());
        if(tm.isEnvironment(tm.environment.PRODUCTION)) {
            // When in production mode rename the files.
            task.pipe(rename({extname: '.min.css'}));
        }
        task.pipe(this.dest(engineOptions.dest));
        task.pipe(done());
        return task;
    },
    engineOptions: {
        src: 'static/fonts/**/*',
        dest: 'built/static/fonts/'
    }
});

As you can see however this is exactly the same format as you building your engine into its own package and it would be more ideal for you to build your own package as mentioned above and do an npm publish on it so that you can get all the benefits of being able to update the package across many many projects easily.

License

Copyright 2017-2019 Ben Blanks

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.