01 August 2018

Konstantin Basharkevich

Web developer

How to set up your very own repository of Node.js modules with Blackjack and version control

ISPsystem currently has 3 front-end teams developing three large projects: ISPmanager for web-server management, VMmanager for virtualization, and BILLmanager for hosting business automation. The teams work simultaneously and on a tight schedule so optimization is a necessary step. To reduce time costs, we use unified solutions and create separate projects for global components. These projects are stored in different repositories and maintained by members of all three teams. The following article is about the repositories setup and our workflow within them.

One of the global modules is an utility for testing user scenarios in our projects. It prepares Docker containers of the Chromium browser with the puppeteer library and runs tests with Mocha. This utility also has some complementary functions that help us solve specific tasks in our projects. Any team member is able to modify the utility according to their needs without bothering each other.

Setup of global projects repositories

We use our own server with GitLab for remote repositories. For us it was essential to keep the workflow that is already familiar to everyone and also keep the opportunity to edit global modules during their development. That’s why we rejected the idea of publishing in private repositories on npmjs.com. Luckily, Node.js modules could be installed not only with NPM but from other sources too, including git repositories.

We develop in TypeScript which is then compiled to JavaScript for further use. But nowadays only the lazy one doesn’t compile their own JavaScript. So we needed to separate storages for the source code and the compiled project.

That means that during development a single feature should be published before the release, in the branch with the same name as the current development branch. In this case we are able to use the experimental version of a module by installing it from the specific branch - the one where we are developing in. It is very convenient for seeing the module in action.

In addition, we create a tag for each publication that stores the current project state. The name of the tag matches the version in package.json. In case of installing from git repository, the tag is specified after hash symbol, for example:

npm install git+ssh://[repository url]#1.0.0

That is how we can specify the module version. There are no worries here that someone could edit something.

There are also tags for unstable versions. But for them we add the shortened commit hash taken from the source code repository where it was published. Here is the example of such tag:

1.0.0_e5541dc1

This technique helps us to have unique tags and also link them to the source code repository.

Speaking of stable and unstable module versions, that’s how we detect them: If the publication is made from master or develop branch, the version is stable; otherwise it is not.

How we organise work with our shared projects

All our agreements would be pointless if we weren’t able to automate the process, the publication process in particular.

Inside package.json utility for tests there is the following command:

“publish:git”: "ts-node ./scripts/publish.ts"
It runs the script stored in the same directory.
import { spawnSync } from 'child_process';
import { mkdirSync, existsSync } from 'fs';
import { join } from 'path';
import chalk from 'chalk';

/**
 * Module publication script
 */



/**
 * Generating parameters for subprocesses
 * @param cwd - subprocess directory
 * @param stdio - input/output settings
 */

const getSpawnOptions = (cwd = process.cwd(), stdio = 'inherit') => ({
  cwd,
  shell: true,
  stdio,
});

/* module directory root */
const rootDir = join(__dirname, '../');

/*checking if there are any uncommitted changes*/
const isDiff = !!spawnSync('git', ['diff'], getSpawnOptions(rootDir, 'pipe')).stdout.toString().trim();
if (isDiff) {
  console.log(chalk.red('There are uncommitted changes'));
} else {
  /* project build*/
  const build = spawnSync('npm', ['run', 'build'], getSpawnOptions(rootDir));
  /* checking build status */
  if (build.status === 0) {
    /* temporary directory for the builds repository*/
    const tempDir = join(rootDir, 'temp');
    if (existsSync(tempDir)) {
      spawnSync('rm', ['-rf', 'temp'], getSpawnOptions(rootDir));
    }
    mkdirSync(tempDir);

    /* getting parameters from the package.json */
    const { name, version, repository } = require(join(rootDir, 'package.json'));
    const originUrl = repository.url.replace(`${name}-source`, name);

    spawnSync('git', ['init'], getSpawnOptions(tempDir));
    spawnSync('git', ['remote', 'add', 'origin', originUrl], getSpawnOptions(tempDir));

    /*current branch name in the module source code repository*/
    const branch = spawnSync(
      'git',
      ['symbolic-ref', '--short', 'HEAD'],
      getSpawnOptions(rootDir, 'pipe')
    ).stdout.toString().trim();
    /*branch name in the builds repository*/
    const buildBranch = branch === 'develop' ? 'master' : branch;

    /* Shortened hash of the last commit in source code repository.
    Used for forming of the unstable version tag */

    const shortSHA = spawnSync(
      'git',
      ['rev-parse', '--short', 'HEAD'],
      getSpawnOptions(rootDir, 'pipe')
    ).stdout.toString().trim();
    /* tag */
    const tag = buildBranch === 'master' ? version : `${version}_${shortSHA}`;

    /* checking if the created tag exists in build repository */
    const isTagExists = !!spawnSync(
      'git',
      ['ls-remote', 'origin', `refs/tags/${tag}`],
      getSpawnOptions(tempDir, 'pipe')
    ).stdout.toString().trim();

    if (isTagExists) {
      console.log(chalk.red(`Tag ${tag} already exists`));
    } else {
      /* checking if branch exists in builds repository */
      const isBranchExits = !!spawnSync(
        'git',
        ['ls-remote', '--exit-code', 'origin', buildBranch],
        getSpawnOptions(tempDir, 'pipe')
      ).stdout.toString().trim();

      if (isBranchExits) {
        /* checkout the target branch */
        spawnSync('git', ['fetch', 'origin', buildBranch], getSpawnOptions(tempDir));
        spawnSync('git', ['checkout', buildBranch], getSpawnOptions(tempDir));
      } else {
        /* checkout master */
        spawnSync('git', ['fetch', 'origin', 'master'], getSpawnOptions(tempDir));
        spawnSync('git', ['checkout', 'master'], getSpawnOptions(tempDir));
        /* creating branch */
        spawnSync('git', ['checkout', '-b', buildBranch], getSpawnOptions(tempDir));
        /* creating initial commit */
        spawnSync('git', ['commit', '--allow-empty', '-m', '"Initial commit"'], getSpawnOptions(tempDir));
      }

      /* removing the old build files */
      spawnSync(
        'rm',
        ['-rf', 'lib', 'package.json', 'package-lock.json', 'README.md'],
        getSpawnOptions(tempDir)
      );

      /* creating build files copies */
      spawnSync('cp', ['-r', 'lib', 'temp/lib'], getSpawnOptions(rootDir));
      spawnSync('cp', ['package.json', 'temp/package.json'], getSpawnOptions(rootDir));
      spawnSync('cp', ['package-lock.json', 'temp/package-lock.json'], getSpawnOptions(rootDir));
      spawnSync('cp', ['README.md', 'temp/README.md'], getSpawnOptions(rootDir));

      /* indexing the build files */
      spawnSync('git', ['add', '--all'], getSpawnOptions(tempDir));

      /* message of the last commit in source code repository */
      const lastCommitMessage = spawnSync(
        'git',
        ['log', '--oneline', '-1'],
        getSpawnOptions(rootDir, 'pipe')
      ).stdout.toString().trim();
      /* commit message in build repository */
      const message = buildBranch === 'master' ? version : lastCommitMessage;

      /* making commit in builds repository */
      spawnSync('git', ['commit', '-m', `"${message}"`], getSpawnOptions(tempDir));

      /* creating a tag in builds repository */
      spawnSync('git', ['tag', tag], getSpawnOptions(tempDir));
      /* pushing changes to the remote repository */
      spawnSync('git', ['push', 'origin', buildBranch], getSpawnOptions(tempDir));
      spawnSync('git', ['push', '--tags'], getSpawnOptions(tempDir));

      console.log(chalk.green('Published successfully!'));
    }

    /* removing temporary directory */
    spawnSync('rm', ['-rf', 'temp'], getSpawnOptions(rootDir));
  } else {
    console.log(chalk.red(`Build was exited exited with code ${build.status}`));
  }
}

console.log(''); // space

The script performs all the necessary commands through Node.js module child_process.

The main stages of the script are:

1. Check for uncommitted changes

const isDiff = !!spawnSync('git', ['diff'], getSpawnOptions(rootDir, 'pipe')).stdout.toString().trim();

Here we check the result of git diff command. It’s not a good thing to add the changes that are absent in the source code into publication. It will also destroy the links between unstable versions and commits.

2. Utility build

const build = spawnSync('npm', ['run', 'build'], getSpawnOptions(rootDir));

Result of the build process goes into build constant. If everything went well, the status parameter would be equal to 0. Otherwise, nothing would be published.

3. Deploying compiled versions repository

The whole deploying process is nothing than pushing changes into particular repository. That's why a script creates a temporary directory inside our project where the git repository is initialised, and it connects it with the remote builds repository.

/* temporary directory for build repository deploy */
const tempDir = join(rootDir, 'temp');
if (existsSync(tempDir)) {
  spawnSync('rm', ['-rf', 'temp'], getSpawnOptions(rootDir));
}
mkdirSync(tempDir);

/* getting parameters from package.json */
const { name, version, repository } = require(join(rootDir, 'package.json'));
const originUrl = repository.url.replace(`${name}-source`, name);

spawnSync('git', ['init'], getSpawnOptions(tempDir));
spawnSync('git', ['remote', 'add', 'origin', originUrl], getSpawnOptions(tempDir));

4. Generating the tag name

First, we should find out the branch name where we perform the publication with the command git symbolic-ref. Then we specify the name of the branch where we would push our changes (in build repository there is no develop branch).

/* current branch name in the module source code repository*/

const branch = spawnSync(
  'git',
  ['symbolic-ref', '--short', 'HEAD'],
  getSpawnOptions(rootDir, 'pipe')
).stdout.toString().trim();

/* branch name in builds repository */
const buildBranch = branch === 'develop' ? 'master' : branch;

By using the command git rev-parse we get a shortened hash of the last commit in our current branch. It would be useful in case of generating unstable version tag name.

/* shortened hash of the last commit in sources repository
used for generating unstable version tag */

const shortSHA = spawnSync(
  'git',
  ['rev-parse', '--short', 'HEAD'],
  getSpawnOptions(rootDir, 'pipe')
).stdout.toString().trim();

And then actually form a tag name.

/* tag */
const tag = buildBranch === 'master' ? version : `${version}_${shortSHA}`;

5. Making sure that the same tag doesn’t exist in the remote repository

/* checking if formed tag exists in builds repository*/
const isTagExists = !!spawnSync(
  'git',
  ['ls-remote', 'origin', `refs/tags/${tag}`],
  getSpawnOptions(tempDir, 'pipe')
).stdout.toString().trim();

If a tag with this name was created earlier, the result of the command git ls-remote wouldn’t be empty. One version should be published only once.

6. Creating a relevant branch in builds repository

As I have mentioned before, the compiled version utility repository is a mirror of the source code repository. That’s why, if the publication isn’t made from the branches master or develop, we shall create the relevant branch in builds repository, or at least make sure that it exists.

/* check if branch exists in builds repository */
const isBranchExits = !!spawnSync(
  'git',
  ['ls-remote', '--exit-code', 'origin', buildBranch],
  getSpawnOptions(tempDir, 'pipe')
).stdout.toString().trim();

if (isBranchExits) {
  /* move to needed branch */
  spawnSync('git', ['fetch', 'origin', buildBranch], getSpawnOptions(tempDir));
  spawnSync('git', ['checkout', buildBranch], getSpawnOptions(tempDir));
} else {
  /* move to master branch */
  spawnSync('git', ['fetch', 'origin', 'master'], getSpawnOptions(tempDir));
  spawnSync('git', ['checkout', 'master'], getSpawnOptions(tempDir));
  /* creating the needed branch */
  spawnSync('git', ['checkout', '-b', buildBranch], getSpawnOptions(tempDir));
  /* creating the initial commit */
  spawnSync('git', ['commit', '--allow-empty', '-m', '"Initial commit"'], getSpawnOptions(tempDir));
}

If the branch hasn’t existed before, we initialize an empty commit with the flag --allow-empty.

7. File preparation

First of all we need to delete everything that might have appeared in the repository because if we use a previously existed branch, it contains the previous utility version.

/* deleting old build files */
spawnSync(
  'rm',
  ['-rf', 'lib', 'package.json', 'package-lock.json', 'README.md'],
  getSpawnOptions(tempDir)
);

Then we move the updated files needed for publication and add them to the repository index.

/* copying build files */
spawnSync('cp', ['-r', 'lib', 'temp/lib'], getSpawnOptions(rootDir));
spawnSync('cp', ['package.json', 'temp/package.json'], getSpawnOptions(rootDir));
spawnSync('cp', ['package-lock.json', 'temp/package-lock.json'], getSpawnOptions(rootDir));
spawnSync('cp', ['README.md', 'temp/README.md'], getSpawnOptions(rootDir));

/* indexing build files */
spawnSync('git', ['add', '--all'], getSpawnOptions(tempDir));

After this manipulation, git recognises all the changes made in the files line by line. That’s how we get the consistent changes history even inside the compiled versions repository.

8. Commit and push changes

We use tag name for stable versions as a commit message in builds repository, and for the unstable ones we utilize the commit message from the source code repository. This way we follow up the idea of the mirror storage.

/* last commit message in source code repository*/
const lastCommitMessage = spawnSync(
  'git',
  ['log', '--oneline', '-1'],
  getSpawnOptions(rootDir, 'pipe')
).stdout.toString().trim();
/* commit message in builds repository */
const message = buildBranch === 'master' ? version : lastCommitMessage;

/* creating commit in builds repository */
spawnSync('git', ['commit', '-m', `"${message}"`], getSpawnOptions(tempDir));

/* creating tag in builds repository */
spawnSync('git', ['tag', tag], getSpawnOptions(tempDir));
/* pushing changes to remote repository */
spawnSync('git', ['push', 'origin', buildBranch], getSpawnOptions(tempDir));
spawnSync('git', ['push', '--tags'], getSpawnOptions(tempDir));

9. Removing the temporary directory

spawnSync('rm', ['-rf', 'temp'], getSpawnOptions(rootDir));

Review of the global project updates

One of the most important actions after global project update is review. Despite the fact that the described technology lets us create absolutely isolated module versions, no one wants to have dozens of different versions of one utility. That’s why each of the common projects should follow a single development path. It’s important to reach an agreement on this point among the teams.

Reviewing the update of common projects is made by the members of all the teams as soon as possible. That is not a trivial task as every team works according to their own sprint and their business varies. Sometimes migration to the new version might take a bit longer.

I could only recommend not to sabotage or postpone this process.

Konstantin Basharkevich

Web developer