mdlink: Linking multiple node modules with a single command

Introduction

As part of the process of working in node.js applications, developers usually use additional libraries and frameworks to ease and speed their development time, and more than often the number of libraries they rely upon is quite high.

It is also not uncommon to need to modify any of those libraries to fix a bug, or fork them to add additional functionality required for the project they are being used in.

The usual way to do this in a node.js project is:

  1. Clone the library's repository in your local machine
  2. cd your library's folder.
  3. sudo npm link to link the module globally.
  4. cd to your project's folder.
  5. npm link <library_name> to link to the library.

If you have multiple libraries that you need to be linking at the same time, or you have to link a library that depends on another library that you may need to link as well, it can be quite cumbersome to link them one by one.

In my free time I developed mdlink. It is a small utility that allows to easily npm link multiple node modules in a given project.

It was born as a reaction of my annoyance while I was developing Ember.js applications with multiple addons at the same time. Per each addon I'd need to clone it and link it. This tool intends to provide an easy way of bulk npm link and removing those links later while not necessary.

Modules are specified in the mdlink.config.json configuration file. e.g:

{
  "base_modules_path": "~/gits/test",
  "modules": {
    "mdlink": {
      "url": "https://github.com/fr0gs/mdlink",
      "path": "~/gits/test/mdlink"
    }
  }
}

Each module to be linked is stated in the modules object. It can have both url & path together, or only one of each.

The base_modules_path is used for the case where there is no path but a url is yet specified. The module will be cloned in base_modules_path/module_name instead.

Let's see an example with a new Ember application. First, download mdlink with npm and create the application:

test-ember:master* λ sudo npm install -g mdlink
test-ember:master* λ ember new test-ember

Then, create a new mdlink.config.json file inside the test-ember app.

test-ember:master* λ mdlink init

And let's say that in our new application we want to locally modify both ember-moment and ember-promise-helpers. We modify the mdlink.config.json file like this:

{
  "base_modules_path": "/home/esteban/gits/test",
  "modules": {
    "ember-moment": {
      "url": "https://github.com/stefanpenner/ember-moment",
      "path": "/home/esteban/gits/test/ember-moment"
    },
    "ember-promise-helpers": {
      "url": "https://github.com/fivetanley/ember-promise-helpers.git",
      "path": "/home/esteban/gits/test/ember-promise-helpers"
    }
  }
}

Do the linking:

test-ember:master* λ mdlink s
[+] Path /home/esteban/gits/test-ember/node_modules/ember-moment already exists, removing it.
[+] <path exists> . Successfully cloned https://github.com/stefanpenner/ember-moment in path: /home/esteban/gits/test/ember-moment
[+] Create link from /usr/lib/node_modules/ember-moment -> /home/esteban/gits/test/ember-moment
-------------------------
[+] <path exists> . Successfully cloned https://github.com/fivetanley/ember-promise-helpers.git in path: /home/esteban/gits/test/ember-promise-helpers
[+] Create link from /usr/lib/node_modules/ember-promise-helpers -> /home/esteban/gits/test/ember-promise-helpers

Verify both modules where properly cloned:

test λ ls -la ~/gits/test/
drwxrwxr-x  4 esteban esteban 4,0K Dez 19 22:07 ./
drwxrwxr-x  8 esteban esteban 4,0K Dez 19 22:05 ../
drwxrwxr-x 10 esteban esteban 4,0K Dez 19 22:36 ember-moment/
drwxrwxr-x  9 esteban esteban 4,0K Dez 19 22:07 ember-promise-helpers/

Modify index.js in both modules, and ember s from the test-ember app folder:

test-ember:master* λ ember s
Linked ember-moment
Linked ember-promise-helpers
Could not start watchman
Visit https://ember-cli.com/user-guide/#watchman for more info.
Livereload server on http://localhost:7020

Build successful (14117ms) – Serving on http://localhost:4200/

Have fun!

Types of dependencies in NPM.

Introduction

When we start a new javascript project and we count on adding a minimal level of complexity to it, we will eventually need to make use of external libraries if we want to avoid writing every feature totally from scratch. For that matter, you can choose to take three approaches:

  • Download the library code bundle and adapt it to your development and deployment process. This is easy if the library is very small, you don't care about updating it's version, and you know very specifically what you want from it.
  • Use a Content Delivery Network to access a remote copy of the library hosted in the cloud.
  • Use a tool like NPM, Bower, or the new kid in town Yarn. This are package managers: pieces of software that allow the developer manage the projet's dependencies by fetching them from a central repository, specify each library's desired version and execute scripts for different steps of the development process.

When you are working on a project and decide to use npm as it's package manager, it will use a file named package.json in the root of the project's directory. This file contains the information about the dependencies on external libraries it has, along with other useful information. It sits as the point of reference that npm will use when fetching and updating your dependencies. More information about the package.json file [here](link to package.json explanation).

How does it work.

Very briefly explained, npm will read the contents of your package.json file and will enumerate the libraries your current project depends on. It will then fetch each library in a recursive manner, namely, each one of those libraries may contain themselves a package.json file enumerating the dependencies that library relies upon, so it'll fetch them again, and so on, and so on.

Dependencies are expressed in a hash with a pair of

{
  "package_name_1": "version range",
  "package_name_2": "version range",
  ...
}

per each of them. The version range can be a string indicating the version range to fetch from the central repository, a tarball or a git url pointing to a specific tag or commit.

There are different ways inside the package.json file to indicate the nature of the dependencies: dependencies, devDependencies and peerDependencies.
You can also find bundledDependencies and optionalDependencies, but I do not see them used that often.

dependencies

These are the libraries your project directly depends on in order to properly execute. There is a direct dependency from the correct functioning of your application to the fetching of this packages. After compiling and minifying your javascript, the final bundle must include these libraries code to work.

devDependencies

They represent the packages needed during the development of your project, such as tools and frameworks, that are not needed in the final product. An example for a project developed with the Ember javascript framework, could be:

"devDependencies": {
  "ember-cli": "2.11.1",
  "ember-cli-sass": "^5.3.1",
}
  • ember-cli is the command line tool that helps the creation of blueprints for the project, run scripts, etc.. much as the rails cli tool.
  • ember-cli-sass contains files to help translate the sass code into css code.

As you can see, a final user of your product is not interested to have this auxiliary tools in the final bundle.

peerDependencies

peerDependencies are used to avoid name clashes and application crashes when a package depends on a library's version that is not the same as the specified one in the project's package.json. For example:

{
  "name": "test_peer",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "grunt-webpack": "^3.0.0",
    "webpack": "^1.0.0"
  }
}

I specify version "1" of webpack and also install grunt-webpack. grunt-webpack states a different webpack version in it's peerDependencies hash:

"peerDependencies": {
  "webpack": "^2.1.0-beta || ^2.2.0-rc || 2 || 3"
},

The result of npm install this project is:

test_peer@1.0.0 /home/esteban/gits/test/test_peer
├─┬ grunt-webpack@3.0.0
│ ├─┬ deep-for-each@1.0.6
│ │ └─┬ is-plain-object@2.0.3
│ │   └── isobject@3.0.0
│ └── lodash@4.17.4
└─┬ UNMET PEER DEPENDENCY webpack@1.0.0

... etc ...

This means that while in my project I downloaded version 1 of webpack, grunt-webpack depends on another set of versions to safely execute, so the package manager will intercede for us and warn us that this setup may break the program.

bundledDependencies

They can also be written as bundleDependencies. They enumerate the dependencies that must be included in your project just as regular dependencies. They will also be included in the tarball generated as the result of executing npm pack.

They are used for example if you want to include third party libraries that are not part of the npm registry, or distribute your project as a module.

optionalDependencies

Just like regular dependencies, but builds don't fail if the dependencies are not met.

That's it, I hope it was a little bit more clarifying for readers!

References: