Backend library for Node.js
[Repository]

Backendjs Documentation

Table of contents

Backend library for Node.js

General purpose backend library. The primary goal is to have a scalable platform for running and managing Node.js servers for Web services implementation.

This project only covers the lower portion of the Web services ecosystem: Node.js processes, HTTP servers, basic API functionality, database access, caching, messaging between processes, metrics and monitoring, a library of tools for developing Node.js servers.

For the UI and presentation layer there are no restrictions what to use as long as it can run on top of the Express server.

Features:

Check out the Documentation for more details.

Installation

To install the module with all optional dependencies if they are available in the system

npm install backendjs

To install from the git

 npm install git+https://github.com/vseryakov/backendjs.git

or simply

 npm install vseryakov/backendjs

Dependencies

Only core required dependencies are installed but there are many modules which require a module to work correctly.

All optional dependencies are listed in the package.json under "modDependencies" so npm cannot use it, only manual install of required modules is supported or it is possible to install all optional dependencies for development purposes.

Here is the list of modules required for each internal feature:

The command below will show all core and optional dependencies, npm install will install only the core dependencies

 bkjs deps -dry-run -mods

Quick start and introduction

or the same using async/await, same methods with a prepended to the name

    > await db.aselect("bk_user", {});
    > await db.aadd("bk_user", { id: 'test2', login: 'test2', secret: 'test2', name' Test 2 name' });
    > await db.aselect("bk_user", { id: 'test2' });

To run an example

Configuration

Almost everything in the backend is configurable using config files, a config database or DNS. The whole principle behind it is that once deployed in production, even quick restarts are impossible to do so there should be a way to push config changes to the processes without restarting.

Every module defines a set of config parameters that defines the behavior of the code, due to the single threaded nature of the Node.js. It is simple to update any config parameter to a new value so the code can operate differently. To achieve this the code must be written in a special way, like driven by configuration which can be changed at any time.

All configuration goes through the configuration process that checks all inputs and produces valid output which is applied to the module variables. Config file or database table with configuration can be loaded on demand or periodically, for example all local config files are watched for modification and reloaded automatically, the config database is loaded periodically which is defined by another config parameter.

Backend runtime

When the backendjs server starts it spawns several processes that perform different tasks.

There are 2 major tasks of the backend that can be run at the same time or in any combination:

These features can be run standalone or under the guard of the monitor which tracks all running processes and restarted any failed ones.

This is the typical output from the ps command on Linux server:

ec2-user    891  0.0  0.6 1071632 49504 ?  Ssl  14:33   0:01 bkjs: monitor
ec2-user    899  0.0  0.6 1073844 52892 ?  Sl   14:33   0:01 bkjs: master
ec2-user    908  0.0  0.8 1081020 68780 ?  Sl   14:33   0:02 bkjs: server
ec2-user    917  0.0  0.7 1072820 59008 ?  Sl   14:33   0:01 bkjs: web
ec2-user    919  0.0  0.7 1072820 60792 ?  Sl   14:33   0:02 bkjs: web
ec2-user    921  0.0  0.7 1072120 40721 ?  Sl   14:33   0:02 bkjs: worker

To enable any task a command line parameter must be provided, it cannot be specified in the config file. The bkjs utility supports several commands that simplify running the backend in different modes.

Application structure

The main purpose of the backendjs is to provide API to access the data, the data can be stored in the database or some other way but the access to that data will be over HTTP and returned back as JSON. This is default functionality but any custom application may return data in whatever format is required.

Basically the backendjs is a Web server with ability to perform data processing using local or remote jobs which can be scheduled similar to Unix cron.

The principle behind the system is that nowadays the API services just return data which Web apps or mobiles apps can render to the user without the backend involved. It does not mean this is simple gateway between the database, in many cases it is but if special processing of the data is needed before sending it to the user, it is possible to do and backendjs provides many convenient helpers and tools for it.

When the API layer is initialized, the api module contains app object which is an Express server.

Special module/namespace app is designated to be used for application development/extension. This module is available in the same way as api and core which makes it easy to refer and extend with additional methods and structures.

The typical structure of a single file backendjs application is the following:

    const bkjs = require('backendjs');
    const api = bkjs.api;
    const app = bkjs.app;
    const db = bkjs.db;

    app.listArg = [];

    // Define the module config parameters
    core.describeArgs('app', [
        { name: "list-arg", array: 1, type: "list", descr: "List of words" },
        { name: "int-arg", type: "int", descr: "An integer parameter" },
     ]);

    // Describe the tables or data models, all DB pools will use it, the master or shell
    // process only creates new tables, workers just use the existing tables
    db.describeTables({
         ...
    });

     // Optionally customize the Express environment, setup MVC routes or else, `api.app` is the Express server
    app.configureMiddleware = function(options, callback)
    {
       ...
       callback()
    }

    // Register API endpoints, i.e. url callbacks
    app.configureWeb = function(options, callback)
    {
        api.app.get('/some/api/endpoint', (req, res) => {
          // to return an error, the message will be translated with internal i18n module if locales
          // are loaded and the request requires it
          api.sendReply(res, err);

          // or with custom status and message, explicitely translated
          api.sendReply(res, 404, res.__({ phrase: "not found", locale: "fr" }));

          // with config check
          if (app.intArg > 5) ...
          if (app.listArg.indexOf(req.query.name) > -1) ...

          // to send data back with optional postprocessing hooks
          api.sendJSON(req, err, data);
          // or simply
          res.json(data);
        });
        ...
        callback();
    }

    // Optionally register post processing of the returned data from the default calls
    api.registerPostProcess('', /^\/account\/([a-z\/]+)$/, (req, res, rows) => { ... });
     ...

    // Optionally register access permissions callbacks
    api.registerAccessCheck('', /^\/test\/list$/, (req, status, callback) => { ...  });
    api.registerPreProcess('', /^\/test\/list$/, (req, status, callback) => { ...  });
     ...
    bkjs.server.start();

Another probably easier way to create single file apps is to use your namespace instead of app:

    const bkjs = require("backendjs");
    const api = bkjs.api;
    const db = bkjs.db;

    const mymod = {
        name: "mymod",
        args: [
            { name: "types", type: "list", descr: "Types allowed" },
            { name: "size", type: "int", descr: "Records in one page" },
        ],
        tables: {
            mytable: {
                id: { type: "int", primary: 1 },
                name: { primary: 2 },
                type: { type: "list" },
                descr: {}
            }
        }
    };
    exports.module = mymod;
    bkjs.core.addModule(mymod);

    mymod.configureWeb = function(options, callback)
    {
        api.app.all("/mymod", async function(req, res) {
            if (!req.query.id) return api.sendReply(res, 400, "id is required");
            req.query.type = mod.types;

            const rows = await db.aselect("mymod", req.query, { ops: { type: "in" }, count: mod.size });
            api.sendJSON(req, null, rows);
        });
    }

    bkjs.server.start();

Except the app.configureWeb and server.start() all other functions are optional, they are here for the sake of completeness of the example. Also because running the backend involves more than just running web server many things can be setup using the configuration options like common access permissions, configuration of the cron jobs so the amount of code to be written to have fully functioning production API server is not that much, basically only request endpoint callbacks must be provided in the application.

As with any Node.js application, node modules are the way to build and extend the functionality, backendjs does not restrict how the application is structured.

Modules

By default no system modules are loaded except bk_user, it must be configured by the -preload-modules config parameter to preload modules from the backendjs/modules/.

Another way to add functionality to the backend is via external modules specific to the backend, these modules are loaded on startup from the backend home subdirectory modules/. The format is the same as for regular Node.js modules and only top level .js files are loaded on the backend startup.

Once loaded they have the same access to the backend as the rest of the code, the only difference is that they reside in the backend home and can be shipped regardless of the npm, node modules and other env setup. These modules are exposed in the core.modules the same way as all other core submodules methods.

Let's assume the modules/ contains file facebook.js which implements custom FB logic:

    const bkjs = require("backendjs");
    const core = bkjs.core;
    const mod = {
        name: "facebook",
        args: [
            { name: "token", descr: "API token" },
        ]
    }
    module.exports = mod;

    mod.configureWeb = function(options, callback) {
       ...
    }

    mod.makeRequest = function(options, callback) {
         core.sendRequest({ url: options.path, query: { access_token: fb.token } }, callback);
    }

This is the main app code:

    const bkjs = require("backendjs");
    const core = bkjs.core;

    // Using facebook module in the main app
    api.app.get("/me", (req, res) => {

       core.modules.facebook.makeRequest({ path: "/me" }, (err, data) => {
          bkjs.api.sendJSON(req, err, data);
       });
    });

    bkj.server.start();

NPM packages as modules

In case different modules is better keep separately for maintenance or development purposes they can be split into separate NPM packages, the structure is the same, modules must be in the modules/ folder and the package must be loadable via require as usual. In most cases just empty index.js is enough. Such modules will not be loaded via require though but by the backendjs core.loadModule machinery, the NPM packages are just keep different module directories separate from each other.

The config parameter preload-packages can be used to specify NPM package names to be loaded separated by comma, as with the default application structure all subfolders inside each NPM package will be added to the core:

If there is a config file present as etc/config it will be loaded as well, this way each package can maintain its default config parameters if necessary without touching other or global configuration. Although such config files will not be reloaded on changes, when NPM installs or updates packages it moves files around so watching the old config is no point because the updated config file will be different.

Database schema definition

The backend support multiple databases and provides the same db layer for access. Common operations are supported and all other specific usage can be achieved by using SQL directly or other query language supported by any particular database. The database operations supported in the unified way provide simple actions like db.get, db.put, db.update, db.del, db.select. The db.query method provides generic access to the database driver and executes given query directly by the db driver, it can be SQL or other driver specific query request.

Before the tables can be queried the schema must be defined and created, the backend db layer provides simple functions to do it:

        db.describeTables({
           album: {
               id: { primary: 1 },                         // Primary key for an album
               name: { pub: 1 },                           // Album name, public column
               mtime: { type: "now" },                     // Modification timestamp
           },
           photo: {
               album_id: { primary: 1 },                   // Combined primary key
               id: { primary: 1 },                         // consisting of album and photo id
               name: { pub: 1, index: 1 },                 // Photo name or description, public column with the index for faster search
               mtime: { type: "now" }
           }
        });

Each database may restrict how the schema is defined and used, the db layer does not provide an artificial layer hiding all specifics, it just provides the same API and syntax, for example, DynamoDB tables must have only hash primary key or combined hash and range key, so when creating table to be used with DynamoDB, only one or two columns can be marked with primary property while for SQL databases the composite primary key can consist of more than 2 columns.

The backendjs always creates several tables in the configured database pools by default, these tables are required to support default API functionality and some are required for backend operations. Refer below for the JavaScript modules documentation that described which tables are created by default. In the custom applications the db.describeTables method can modify columns in the default table and add more columns if needed.

For example, to make age and some other columns in the accounts table public and visible by other users with additional columns the following can be done in the api.initApplication method. It will extend the bk_user table and the application can use new columns the same way as the already existing columns. Using the birthday column we make 'age' property automatically calculated and visible in the result, this is done by the internal method api.processAccountRow which is registered as post process callback for the bk_user table. The computed property age will be returned because it is not present in the table definition and all properties not defined and configured are passed as is.

The cleanup of the public columns is done by the api.sendJSON which is used by all API routes when ready to send data back to the client. If any post-process hooks are registered and return data itself then it is the hook responsibility to cleanup non-public columns.

    db.describeTables({
        bk_user: {
            birthday: {},
            ssn: {},
            salary: { type: "int" },
            occupation: {},
            home_phone: {},
            work_phone: {},
        });

    app.configureWeb = function(options, callback)
    {
       db.setProcessRow("post", "bk_user", this.processAccountRow);
       ...
       callback();
    }
    app.processAccountRow = function(req, row, options)
    {
       if (row.birthday) row.age = Math.floor((Date.now() - core.toDate(row.birthday))/(86400000*365));
    }

To define tables inside a module just provide a tables property in the module object, it will be picked up by database initialization automatically.

    const mod = {
        name: "billing",
        tables: {
            invoices: {
                id: { type: "int", primary: 1 },
                name: {},
                price: { type: "real" },
                mtime: { type: "now" }
            }
        }
    }
    module.exports = mod;

    // Run db setup once all the DB pools are configured, for example produce dynamic icon property
    // for each record retrieved
    mod.configureModule = function(options, callback)
    {
        db.setProcessRows("post", "invoices", function(req, row, opts) {
         if (row.id) row.icon = "/images/" + row.id + ".png";
     });
        callback();
    }

Tables can have aliases

This is useful for easier naming conventions or switching to a different table name on the fly without changinbf the code, access to the table by it is real name is always available.

For example:

bksh -db-aliases-bk_user users

> await db.aget("bk_user", { login: "u1" })
> { login: "u1", name: "user", .... }

> await db.aget("users", { login: "u1" })
> { login: "u1", name: "user", .... }

API requests handling

All methods will put input parameters in the req.query, GET or POST.

One way to verify input values is to use lib.toParams, only specified parameters will be returned and converted according to the type or ignored.

Example:

   var params = {
      test1: { id: { type: "text" },
               count: { type: "int" },
               email: { regexp: /^[^@]+@[^@]+$/ }
      }
   };

   api.app.all("/endpoint/test1", function(req, res) {
      const query = lib.toParams(req.query, params.test1);
      if (typeof query == "string") return api.sendReply(res, 400, query);
      ...
   });

Example of TODO application

Here is an example how to create simple TODO application using any database supported by the backend. It supports basic operations like add/update/delete a record, show all records.

Create a file named app.js with the code below.

    const bkjs = require('backendjs');
    const api = bkjs.api;
    const lib = bkjs.lib;
    const app = bkjs.app;
    const db = bkjs.db;

    // Describe the table to store todo records
    db.describeTables({
       todo: {
           id: { type: "uuid", primary: 1 },  // Store unique task id
           due: {},                           // Due date
           name: {},                          // Short task name
           descr: {},                         // Full description
           mtime: { type: "now" }             // Last update time in ms
       }
    });

    // API routes
    app.configureWeb = function(options, callback)
    {
        api.app.get(/^\/todo\/([a-z]+)$/, async function(req, res) {
           var options = api.getOptions(req);
           switch (req.params[0]) {
             case "get":
                if (!req.query.id) return api.sendReply(res, 400, "id is required");
                const rows = await db.aget("todo", { id: req.query.id }, options);
                api.sendJSON(req, null, rows);
                break;

             case "select":
                options.noscan = 0; // Allow empty scan of the whole table if no query is given, disabled by default
                const rows = await db.aselect("todo", req.query, options);
                api.sendJSON(req, null, rows);
                break;

            case "add":
                if (!req.query.name) return api.sendReply(res, 400, "name is required");
                // By default due date is tomorrow
                if (req.query.due) req.query.due = lib.toDate(req.query.due, Date.now() + 86400000).toISOString();
                db.add("todo", req.query, options, (err, rows) => {
                    api.sendJSON(req, err, rows);
                });
                break;

            case "update":
                if (!req.query.id) return api.sendReply(res, 400, "id is required");
                const rows = await db.aupdate("todo", req.query, options);
                api.sendJSON(req, null, rows);
                break;

            case "del":
                if (!req.query.id) return api.sendReply(res, 400, "id is required");
                db.del("todo", { id: req.query.id }, options, (err, rows) => {
                    api.sendJSON(req, err, rows);
                });
                break;
            }
        });
        callback();
     }
     bkjs.server.start();

Now run it with an option to allow API access without an account:

node app.js -log debug -web -api-allow-path /todo -db-create-tables

To use a different database, for example PostgresSQL(running localy) or DynamoDB(assuming EC2 instance), all config parametetrs can be stored in the etc/config as well

node app.js -log debug -web -api-allow-path /todo -db-pool dynamodb -db-dynamodb-pool default -db-create-tables
node app.js -log debug -web -api-allow-path /todo -db-pool pg -db-pg-pool default -db-create-tables

API commands can be executed in the browser or using curl:

curl 'http://localhost:8000/todo?name=TestTask1&descr=Descr1&due=2015-01-01`
curl 'http://localhost:8000/todo/select'

Backend directory structure

When the backend server starts and no -home argument passed in the command line the backend makes its home environment in the ~/.bkjs directory. It is also possible to set the default home using BKJS_HOME environment variable.

The backend directory structure is the following:

Environment variables

On startup some env variable will be used for initial configuration:

Cache configurations

Database layer support caching of the responses using db.getCached call, it retrieves exactly one record from the configured cache, if no record exists it will pull it from the database and on success will store it in the cache before returning to the client. When dealing with cached records, there is a special option that must be passed to all put/update/del database methods in order to clear local cache, so next time the record will be retrieved with new changes from the database and refresh the cache, that is { cached: true } can be passed in the options parameter for the db methods that may modify records with cached contents. In any case it is required to clear cache manually there is db.clearCache method for that.

Also there is a configuration option -db-caching to make any table automatically cached for all requests.

Local

If no cache is configured the local driver is used, it keeps the cache on the master process in the LRU pool and any worker or Web process communicate with it via internal messaging provided by the cluster module. This works only for a single server.

Redis

Set ipc-client=redis://HOST[:PORT] that points to the server running Redis server.

The config option max_attempts defines maximum number of times to reconnect before giving up. Any other node-redis module parameter can be passed as well in the options or url, the system supports special parameters that start with bk-, it will extract them into options automatically.

For example:

ipc-client=redis://host1?bk-max_attempts=3
ipc-client-backup=redis://host2
ipc-client-backup-options-max_attempts=3

PUB/SUB or Queue configurations

Redis system bus

If configured all processes subscribe to it and listen for system messages, it must support PUB/SUB and does not need to be reliable. Websockets in the API server also use the system bus to send broadcasts between multiple api instances.

ipc-client-system=redis://
ipc-system-queue=system

Redis Queue

To configure the backend to use Redis for job processing set ipc-queue=redis://HOST where HOST is IP address or hostname of the single Redis server. This driver implements reliable Redis queue, with visibilityTimeout config option works similar to AWS SQS.

Once configured, then all calls to jobs.submitJob will push jobs to be executed to the Redis queue, starting somewhere a backend master process with -jobs-workers 2 will launch 2 worker processes which will start pulling jobs from the queue and execute.

The naming convention is that any function defined as function(options, callback) can be used as a job to be executed in one of the worker processes.

An example of how to perform jobs in the API routes:


    core.describeArgs('app', [
        { name: "queue", descr: "Queue for jobs" },
    ]);
    app.queue = "somequeue";

    app.processAccounts = function(options, callback) {
        db.select("bk_user", { type: options.type || "user" }, (err, rows) => {
          ...
          callback();
        });
    }

    api.all("/process/accounts", (req, res) => {
        jobs.submitJob({ job: { "app.processAccounts": { type: req.query.type } } }, { queueName: app.queue }, (err) => {
            api.sendReply(res, err);
        });
    });

SQS

To use AWS SQS for job processing set ipc-queue=https://sqs.amazonaws.com...., this queue system will poll SQS for new messages on a worker and after successful execution will delete the message. For long running jobs it will automatically extend visibility timeout if it is configured.

Local

The local queue is implemented on the master process as a list, communication is done via local sockets between the master and workers. This is intended for a single server development purposes only.

NATS

To use NATS (https://nats.io) configure a queue like ipc-queue-nats=nats://HOST:PORT, it supports broadcasts and job queues only, visibility timeout is supported as well.

RabbitMQ

To configure the backend to use RabbitMQ for messaging set ipc-queue=amqp://HOST and optionally amqp-options=JSON with options to the amqp module. Additional objects from the config JSON are used for specific AMQP functions: { queueParams: {}, subscribeParams: {}, publishParams: {} }. These will be passed to the corresponding AMQP methods: amqp.queue, amqp.queue.subcribe, amqp.publish. See AMQP Node.js module for more info.

Security configurations

API only

This is default setup of the backend when all API requests except must provide valid signature and all HTML, JavaScript, CSS and image files are available to everyone. This mode assumes that Web development will be based on 'single-page' design when only data is requested from the Web server and all rendering is done using JavaScript. This is how the examples/api/api.html developers console is implemented, using JQuery-UI and Knockout.js.

To see current default config parameters run any of the following commands:

    bkjs bkhelp | grep api-allow

    node -e 'require("backendjs").core.showHelp()'

Secure Web site, client verification

This is a mode when the whole Web site is secure by default, even access to the HTML files must be authenticated. In this mode the pages must defined 'Backend.session = true' during the initialization on every html page, it will enable Web sessions for the site and then no need to sign every API request.

The typical client JavaScript verification for the html page may look like this, it will redirect to login page if needed, this assumes the default path '/public' still allowed without the signature:

   <link href="/css/bkjs.bundle.css" rel="stylesheet">
   <script src="/js/bkjs.bundle.js" type="text/javascript"></script>
   <script>
    $(function () {
       bkjs.session = true;
       $(bkjs).on("bkjs.nologin", function() { window.location='/public/index.html'; });
       bkjs.koInit();
   });
   </script>

Secure Web site, backend verification

On the backend side in your application app.js it needs more secure settings defined i.e. no html except /public will be accessible and in case of error will be redirected to the login page by the server. Note, in the login page bkjs.session must be set to true for all html pages to work after login without singing every API request.

  1. We disable all allowed paths to the html and registration:
   app.configureMiddleware = function(options, callback) {
       this.allow.splice(this.allow.indexOf('^/$'), 1);
       this.allow.splice(this.allow.indexOf('\\.html$'), 1);
       callback();
   }
  1. We define an auth callback in the app and redirect to login if the request has no valid signature, we check all html pages, all allowed html pages from the /public will never end up in this callback because it is called after the signature check but allowed pages are served before that:
   api.registerPreProcess('', /^\/$|\.html$/, (req, status, callback) => {
       if (status.status != 200) {
           status.status = 302;
           status.url = '/public/index.html';
       }
       callback(status);
   });

WebSockets connections

The simplest way is to configure ws-port to the same value as the HTTP port. This will run WebSockets server along the regular Web server.

In the browser the connection config is stored in the bkjs.wsconf and by default it connects to the local server on port 8000.

There are two ways to send messages via Websockets to the server from a browser:

    bkjs.wsConnect({ path: "/project/ws?id=1" });

    $(bkjs).on("bkjs.ws.message", (msg) => {
        switch (msg.op) {
        case "/account/update":
            bkjs.wsSend("/account/ws/account");
            break;

        case "/project/update":
            for (const p in msg.project) app.project[p] = msg.project[p];
            break;

        case "/message/new":
            bkjs.showAlert("info", `New message: ${msg.msg}`);
            break;
        }
    });

    // Notify all clients who is using the project being updated
    api.app.all("/project/ws", (req, res) => {
        switch (req.query.op) {
        case "/project/update":
           //  some code ....
           api.wsNotify({ query: { id: req.query.project.id } }, { op: "/project/update", project: req.query.project });
           break;
       }
       res.send("");
    });

In any case all Websocket messages sent from the server will arrive in the event handler and must be formatted properly in order to distinguish what is what, this is the application logic. If the server needs to send a message to all or some specific clients for example due to some updates in the DB, it must use the api.wsNotify function.

    // Received a new message for a user from external API service, notify all websocket clients by account id
    api.app.post("/api/message", (req, res) => {
        ....
        ... processing logic
        ....
        api.wsNotify({ account_id: req.query.uid }, { op: "/message/new", msg: req.query.msg });
    });

Versioning

There is no ready to use support for different versions of API because there is no just one solution that satisfies all applications. But there are tools ready to use that will allow to implement such versioning system in the backend. Some examples are provided below:

    api.all(/\/domain\/(get|put|del)/, function(req, res) {
        var options = api.getOptions(req);
        var cmd = req.params[0];
        if (options.apiVersion) cmd += "/" + options.apiVersion;
        switch (cmd) {
        case "get":
            break;

        case "get/2015-01-01":
            break;

        case "put":
            break;

        case "put/2015-02-01":
            break;

        case "del"
            break;
        }
    });

    var options = api.getOptions(req);
    var version = lib.toVersion(options.appVersion);
    switch (req.params[0]) {
    case "get":
        if (version < lib.toVersion("1.2.5")) {
            res.json({ id: 1, name: "name", description: "descr" });
            break;
        }
        if (version < lib.toVersion("1.1")) {
            res.json([id, name]);
            break;
        }
        res.json({ id: 1, name: "name", descr: "descr" });
        break;
    }

The actual implementation can be modularized, split into functions, controllers.... there are no restrictions how to build the working backend code, the backend just provides all necessary information for the middleware modules.

The backend tool: bkjs

The purpose of the bkjs shell script is to act as a helper tool in configuring and managing the backend environment and as well to be used in operations on production systems. It is not required for the backend operations and provided as a convenience tool which is used in the backend development and can be useful for others running or testing the backend.

Run bkjs help to see description of all available commands.

The tool is multi-command utility where the first argument is the command to be executed with optional additional arguments if needed.

On Linux, when started the bkjs tries to load and source the following global config files:

    /etc/conf.d/bkjs
    /etc/sysconfig/bkjs

Then it try to source all local config files:

    $BKJS_HOME/etc/profile
    $BKJS_HOME/etc/profile.local

Any of the following config files can redefine any environment variable thus pointing to the correct backend environment directory or customize the running environment, these should be regular shell scripts using bash syntax.

To check all env variables inside bkjs just run the command bkjs env

The tool provides some simple functions to parse comamndline arguments, the convention is that argument name must start with a single dash followed by a value.

Extending bkjs

The utility is extended via external scripts that reside in the tools/ folders.

When bkjs is running it treats the first arg as a command:

if no internal commands match it starts loading external scripts that match with bkjs-PART1-* where PART1 is the first part of the command before first dash.

For example, when called:

bkjs ec2-check-hostname

it will check the command in main bkjs cript, not found it will search for all files that match bkjs-ec2-* in all known folders.

The file are loaded from following directories in this particular order:

BKJS_DIR always points to the backendjs installation directory.

BLKJS_TOOLS env variable may contain a list of directories separated by spaces, this variable or command line arg -tools is the way to add custom commands to bkjs. BKJS_TOOLS var is usually set in one of the profile config files mentioned above.

Example of a typical bkjs command:

We need to set BKJS_TOOLS to point to our package(s), on Darwin add it to ~/.bkjs/etc/profile as

BKJS_TOOLS="$HOME/src/node-pkg/tools"

Create a file $HOME/tools/bkjs-super

#!/bin/sh

case "$BKJS_CMD" in
  super)
   arg1=$(get_arg -arg1)
   arg2=$(get_arg -arg1 1)
   [ -z $arg1 ] && echo "-arg1 is required" && exit 1
   ...
   exit

  super-all)
   ...
   exit
   ;;

  help)
   echo ""
   echo "$0 super -arg1 ARG -arg2 ARG ..."
   echo "$0 super-all ...."
   ;;
esac

Now calling bkjs super or bkjs super-all will use the new $HOME/tools/bkjs-super file.

Web development notes

Then run the dev build script to produce web/js/bkjs.bundle.js and web/css/bkjs.bundle.css

cd node_modules/backendjs && npm run devbuild

Now instead of including a bunch of .js or css files in the html pages it only needs /js/bkjs.bundle.js and /css/bkjs.bundle.css. The configuration is in the package.json file.

The list of files to be used in bundles is in the package.json under config.bundles.

To enable auto bundler in your project just add to the local config ~/.bkjs/etc/config.local a list of directories to be watched for changes. For example adding these lines to the local config will enable the watcher and bundle support

watch-web=web/js,web/css,$HOME/src/js,$HOME/src/css
watch-ignore=.bundle.(js|css)$
watch-build=bkjs bundle -dev

The simple script below allows to build the bundle and refresh Chrome tab automatically, saves several clicks:

#!/bin/sh
bkjs bundle -dev -file $2
[ "$?" != "0" ] && exit
osascript -e "tell application \"Google Chrome\" to reload (tabs of window 1 whose URL contains \"$1\")"

To use it call this script instead in the config.local:

watch-build=bundle.sh /website

NOTE: Because the rebuild happens while the watcher is running there are cases like the server is restarting or pulling a large update from the repository when the bundle build may not be called or called too early. To force rebuild run the command:

bkjs bundle -dev -all -force

Deployment use cases

AWS instance setup with node and backendjs

NOTE: if running behind a Load balancer and actual IP address is needed set Express option in the command line -api-express-options {"trust%20proxy":1}. In the config file replacing spaces with %20 is not required.

AWS Provisioning examples

Make an AMI

On the running machine which will be used for an image:

bksh -aws-create-image -no-reboot

Use an instance by tag for an image:

bksh -aws-create-image -no-reboot -instance-id `bkjs ec2-show -tag api -fmt id | head -1`

Update Route53 with all IPs from running instances

bksh -aws-set-route53 -name elasticsearch.ec-internal -filter elasticsearch

Configure HTTP port

The first thing when deploying the backend into production is to change API HTTP port, by default is is 8000, but we would want port 80 so regardless how the environment is setup it is ultimately 2 ways to specify the port for HTTP server to use:

Backend library development (Mac OS X, developers)

Simple testing facility

Included a simple testing tool, it is used for internal bkjs testing but can be used for other applications as well.

The convention is to create a test file in the tests/ folder, each test file can define one or more test functions named in the form tests.test_NAME where NAME is any custom name for the test, for example:

File tests/example.js:

tests.test_example = function(callback)
{
    expect(1 == 2, "expect 1 eq 2")

    callback();
}

Then to run all tests

bkjs test-all

More details are in the documentation or doc.html

API endpoints provided by the backend

All API endpoints are optional and can be disabled or replaced easily. By default the naming convention is:

 /namespace/command[/subname[/subcommand]]

Any HTTP methods can be used because its the command in the URL that defines the operation. The payload can be url-encoded query parameters or JSON or any other format supported by any particular endpoint. This makes the backend universal and usable with any environment, not just a Web browser. Request signature can be passed in the query so it does not require HTTP headers at all.

Authentication and sessions

Signature

All requests to the API server must be signed with account login/secret pair.

The resulting signature is sent as HTTP header bk-signature or in the header specified by the api-signature-name config parameter.

For JSON content type, the method must be POST and no query parameters specified, instead everything should be inside the JSON object which is placed in the body of the request. For additional safety, SHA1 checksum of the JSON payload can be calculated and passed in the signature, this is the only way to ensure the body is not modified when not using query parameters.

See web/js/bkjs.js function bkjs.createSignature or api.js function api.createSignature for the JavaScript implementations.

Authentication API

    $.ajax({ url: "/login?login=test123&secret=test123&_session=1",
        success: function(json, status, xhr) { console.log(json) }
    });

    > { id: "XXXX...", name: "Test User", login: "test123", ...}

Accounts

The accounts API manages accounts and authentication, it provides basic user account features with common fields like email, name, address.

Health enquiry

When running with AWS load balancer there should be a url that a load balancer polls all the time and this must be very quick and lightweight request. For this purpose there is an API endpoint /ping that just responds with status 200. It is open by default in the default api-allow-path config parameter.

Data

The data API is a generic way to access any table in the database with common operations, as oppose to the any specific APIs above this API only deals with one table and one record without maintaining any other features like auto counters, cache...

Because it exposes the whole database to anybody who has a login it is a good idea to disable this endpoint in the production or provide access callback that verifies who can access it.

This is implemented by the data module from the core.

System API

The system API returns information about the backend statistics, allows provisioning and configuration commands and other internal maintenance functions. By default is is open for access to all users but same security considerations apply here as for the Data API.

This is implemented by the system module from the core. To enable this functionality specify -preload-modules=bk_system.

Author

Vlad Seryakov

Check out the Documentation for more details.

Module: api

HTTP API to the server from the clients, this module implements the basic HTTP(S) API functionality with some common features. The API module incorporates the Express server which is exposed as api.app object, the master server spawns Web workers which perform actual operations and monitors the worker processes if they die and restart them automatically. How many processes to spawn can be configured via -server-max-workers config parameter.

When an HTTP request arrives it goes over Express middleware, but before processing any registered routes there are several steps performed:

Module: app

Author: Vlad Seryakov vseryakov@gmail.com backendjs 2018

This is a skeleton module to be extended by the specific application logic. It provides all callbacks and hooks that are called by the core backend modules during different phases, like initialization, shutting down, etc...

It should be used for custom functions and methods to be defined, the app module is always available.

All app modules in the modules/ subdirectory use the same prototype, i.e. all hooks are available for custom app modules as well.

Module: auth

Module: aws

AWS Cloud API interface

Module: core

The primary object containing all config options and common functions

Module: db

The Database API, a thin abstraction layer on top of SQLite, PostgreSQL, DynamoDB and Cassandra. The idea is not to introduce new abstraction layer on top of all databases but to make the API usable for common use cases. On the source code level access to all databases will be possible using this API but any specific usage like SQL queries syntax or data types available only for some databases will not be unified or automatically converted but passed to the database directly. Only conversion between JavaScript types and database types is unified to some degree meaning JavaScript data type will be converted into the corresponding data type supported by any particular database and vice versa.

Basic operations are supported for all database and modelled after NoSQL usage, this means no SQL joins are supported by the API, only single table access. SQL joins can be passed as SQL statements directly to the database using low level db.query API call, all high level operations like add/put/del perform SQL generation for single table on the fly.

The common convention is to pass options object with flags that are common for all drivers along with specific, this options object can be modified with new properties but all driver should try not to modify or delete existing properties, so the same options object can be reused in subsequent operations.

All queries and update operations ignore properties that starts with underscore.

Before the DB functions can be used the core.init MUST be called first, the typical usage:

       var backend = require("backendjs"), core = backend.core, db = backend.db;
       core.init(function(err) {
           db.add(...
           ...
       });

All database methods can use default db pool or any other available db pool by using pool: name in the options. If not specified, then default db pool is used, sqlite is default if no -db-pool config parameter specified in the command line or the config file. Even if the specified pool does not exist, the default pool will be returned, this allows to pre-confgure the app with different pools in the code and enable or disable any particular pool at any time.

Example, use PostgreSQL db pool to get a record and update the current pool

       db.get("bk_user", { login: "123" }, { pool: "pg" }, (err, row) => {
           if (row) db.update("bk_user", row);
       });

       const user = await db.aget("bk_user", { login: "123" });

Most database pools can be configured with options min and max for number of connections to be maintained, so no overload will happen and keep warm connection for faster responses. Even for DynamoDB which uses HTTPS this can be configured without hitting provisioned limits which will return an error but put extra requests into the waiting queue and execute once some requests finished.

Example:

       db-pg-pool-max = 100
       db-dynamodb-pool-max = 100

Also, to spread functionality between different databases it is possible to assign some tables to the specific pools using db-X-pool-tables parameters thus redirecting the requests to one or another databases depending on the table, this for example can be useful when using fast but expensive database like DynamoDB for real-time requests and slower SQL database running on some slow instance for rare requests, reports or statistics processing.

Example, run the backend with default PostgreSQL database but keep all config parametrs in the DynamoDB table for availability:

       db-pool = pg
       db-dynamodb-pool = default
       db-dynamodb-pool-tables = bk_config

The following databases are supported with the basic db API methods: Sqlite, PostgreSQL, DynamoDB, Elasticsearch

Multiple connections of the same type can be opened, just add N suffix to all database config parameters where N is a number, referer to such pools in the code as poolN or by an alias.

Example:

       db-sqlite1-pool = billing
       db-sqlite1-pool-max = 10
       db-sqlite1-pool-options-path = /data/db
       db-sqlite1-pool-options-journal_mode = OFF
       db-sqlite1-pool-alias = billing

       in the Javascript:

       db.select("bills", { status: "ok" }, { pool: "billing" }, lib.log)
       await db.aselect("bills", { status: "ok" }, { pool: "billing" })

Create a database pool that works with ElasticSearch server, only the hostname and port will be used, by default each table is stored in its own index.

To define shards and replicas per index:

To support multiple seed nodes a parameter -db-elasticsearch-pool-options-servers=1.1.1.1,2.2.2.2 can be specified, if the primary node fails it will switch to other configured nodes. To control the switch retries and timeout there are options:

On successful connect to any node the driver retrieves full list of nodes in the cluster and switches to a random node, this happens every discovery-interval in milliseconds, default is 1h, it can be specified as -db-elasticserch-pool-options-discovery-interval=300000

Module: events

Event queue processor

If any of events-worker-queue-XXX parameters are defined then workers subscribe to configured event queues and listen for events.

Each event queue can run multiple functions idependently but will ack/nack for all functions so to deal with replay dups it is advised to split between multiple consumers using the syntax: queue#channel@consumer

Multiple event queues can be defined and processed at the same time.

An event processing function takes 2 arguments, an event and callback to call on finish

Module: httpget

Downloads file using HTTP and pass it to the callback if provided

On end, the object params will contain the following updated properties:

Note: SIDE EFFECT: the params object is modified in place so many options will be changed/removed or added

Module: ipc

IPC communications between processes and support for caching and subscriptions via queues.

The module is EventEmitter and emits messages received.

Some drivers may support TTL so global options.ttl or local options.ttl can be used for put/incr operations and it will honored if it is suported.

For caches that support maps, like Redis or Hazelcast the options.mapName can be used with get/put/incr/del to work with maps and individual keys inside maps.

All methods use options.queueName or options.cacheName for non-default queue or cache. If it is an array then a client will be picked sequentially by maintaining internal sequence number.

To specify a channel within a queue use the format queueName#channelName, for drivers that support multiple channels like NATS/Redis the channel will be used for another subscription within the same connection.

For drivers (NATS) that support multiple consumers the full queue syntax is queueName#channelName@groupName or queueName@groupName, as well as the groupName property in the subscribe options.

A special system queue can be configured and it will be used by all processes to listen for messages on the channel bkjs:role, where the role is the process role, the same messages that are processed by the server/worker message handlers like api:restart, config:init,.... All instances will be listening and processing these messages at once, the most usefull use case is refreshing the DB config on demand or restarting without configuring any other means like SSH, keys....

Queue client using RabbitMQ server

To enable install the npm module:

   npm i -g amqplib

Module: nats

Queue client using NATS server

To enable install the npm modules:

   npm i -g nats

Configuration:

 ipc-queue-nats=nats://localhost:4222

Cache/queue client based on Redis server using https://github.com/NodeRedis/node_redis3

The queue client implements reliable queue using sorted sets, one for the new messages and one for the messages that are being processed if timeout is provided. With the timeout this queue works similar to AWS SQS.

The interval config property defines in ms how often to check for new messages after processing a message, i.e. after a messages processed it can poll immediately or after this amount of time

The retryInterval config property defines in ms how often to check for new messages after an error or no data, i.e. on empty pool when no messages are processed it can poll immediately or after this amount of time

The visibilityTimeout property specifies to use a shadow queue where all messages that are being processed are stored, while the message is processed the timestamp will be updated so the message stays in the queue, if a worker exists or crashes without confirming the message finished it will be put back into the work queue after visibilityTimeout milliseconds. The queue name that keeps active messages is appended with #.

Protocol rediss: will use TLS to connect to Redis servers, this is required for RedisCche Serverless

The threshold property defines the upper limit of how many active messages can be in the queue when to show an error message, this is for monitoring queue performance

The rate limiter implementes Tocken Bucket algorithm using Lua script inside Redis, the only requirement is that all workers to use NTP for time synchronization

Examples:

   ipc-client=redis://host1
   ipc-client-options-interval=1000
   ipc-client=redis://host1?bk-visibilityTimeout=30000&bk-count=2

Queue client using AWS SQS, full queue url can be used or just the name as sqs://queuename

The count config property specifies how messages to process at the same time, default is 1.

The interval config property defines in ms how often to check for new messages after processing a message, i.e. after a messages processed it can poll immediately or after this amount of time, default is 1000 milliseconds.

The retryInterval config property defines in ms how often to check for new messages after an error or no data, i.e. on empty pool when no messages are processed it can poll immediately or after this amount of time, default is 5000 mulliseconds.

The visibilityTimeout property specifies how long the messages being processed stay hidden, in milliseconds.

The timeout property defines how long to wait for new messages, i.e. the long poll, in milliseconds

The retryCount and retryTimeout define how many times to retry failed AWS HTTP requests, default is 5 times starting with the backoff starting at 500 milliseconds.

For messages that have startTime property which is the time in the future when a message must be actually processed there is a parameter maxTimeout which defines in milliseconds the max time a messsage can stay invisible while waiting for its scheduled date, default is 6 hours, the AWS max is 12 hours. The scheduling is implemented using AWS visibilityTimeout feature, keep scheduled messages hidden until the actual time.

Examples:

   ipc-queue=sqs://messages?bk-interval=60000
   ipc-queue=https://sqs.us-east-1.amazonaws.com/123456/messages?bk-visibilityTimeout=300&bk-count=2

Module: jobs

Job queue processor

When launched with jobs-workers parameter equal or greater than 0, the master spawns a number of workers which subscribe to configured job queues or the default queue and listen for messsges. A job message is an object that defines what method from which module to run with the options as the first argument and a callback as the second.

Multiple job queues can be defined and processed at the same time.

Module: lib

Common utilities and useful functions

Combined parser with type validation

Module: logger

Simple logger utility for debugging

Module: syslog

Module: metrics

Module: msg

Messaging and push notifications for mobile and other clients, supports Apple, Google and AWS/SNS push notifications.

Emits a signal uninstall(client, device_id, account_id) on device invalidation or if a device token is invalid as reported by the server, account_id may not be available.

Module: pool

Create a resource pool, create and close callbacks must be given which perform allocation and deallocation of the resources like db connections.

Options defines the following properties:

If no create implementation callback is given then all operations are basically noop but still cals the callbacks.

Example: var pool = new Pool({ min: 1, max: 5, create: function(cb) { someDb.connect(function(err) { cb(err, this) } }, destroy: function(client) { client.close() } })

     pool.aquire(function(err, client) {
        ...
        client.findItem....
        ...
        pool.release(client);

     });

Module: run

Module: server

The main server class that starts various processes

Module: shell

Shell command interface for bksh

This module is supposed to be extended with commands, the format is `shell.cmdNAME``

where NAME is he commnd name in camel case

For example:

 const bkjs = require("backendjs");
 const shell = bkjs.shell;
 
 shell.cmdMyCommand = function(options) { console.log("hello"); return "continue" }

Now if i call bksh -my-command it will print hello and launch the repl, instead of retuning continue if the command must exit jut call process.exit()

Run bksh -shell-help to see all registered shell commands

Module: watch

Watch the sources for changes and restart the server

Module: bk_data

Account management

Module: bk_system

System management

Module: bk_user

Account management