Building stable CLI apps with Dart

It looked fine, until I sat on it

Dart with the DCli console SDK for Dart provides excellent tooling for building CLI apps.

In this plog - a blog with a poem - we are going to look at the problems of deploying stable Dart CLI apps.

If you are already coding in Dart then it makes a lot of sense to use Dart in your build environment.

At OnePub, our entire build, CI/CD and production systems run on Dart CLI tooling.

We use the 'configuration as code' approach in our environments and our build system is capable of spooling up a VM and building a production ready node without human intervention.

Given how heavily we depend on Dart CLI apps, it's critical that we can deploy them in a manner that ensures they are stable.

I create enough bugs all by myself without the build system chipping in.

Distributing CLI apps using the Dart team's standard recommendations will result in an unstable app.

scepticism is healthy

When building CLI apps within the Dart ecosystem, the standard way to install them is via a Dart repository such as pub.dev or OnePub.

CLI apps installed in this way are essentially distributed as source code. Deploying a CLI app to a system as a simple as:

publish the CLI app

cd mycliapp
dart pub publish

install the app on a target system.

dart pub global activate <my cli app>

If your CLI apps are part of your corporate intellectual property (e.g. you don't want to release them to the world) then you can use a OnePub private repository.

The publish/install process is almost identical

publish the app to OnePub (into your own private repository)

cd mycliapp
onepub pub private
dart pub publish

Install the app from OnePub

onepub global activate <my cli app>

Whether you use pub.dev or OnePub to install your app, the action of activation triggers the Dart SDK to compile your app on the target system.

This process works pretty well, as the Dart SDK includes everything you require to compile a Dart CLI app.

However there are some pitfalls, with the main source of instability coming from Dart's semantic versioning system.

The problem with that chair

Let's have a look at an example:

name: mycliapp
environment: 
  sdk: '>=2.17.0 <3.0.0'
dependencies:
    path: ^1.0.0
    dcli: ^1.35.0
    args: ^1.0.0

The above is an example of a fairly typical pubspec.yaml.

The key to understanding our problem is the hat '^' symbol used in the dependency versions.

For clarity, the hat '^' in front of a version, such as '^1.0.0', means that you expect that your app 'mycliapp' will work with any version of the package from version 1.0.0 up to but excluding 2.0.0.

The Dart team recommends that you allow a range of versions in your dependency list, as it makes it easy when working with transitive dependencies (Your app depends on DCli, DCli depends on path so path is a transitive dependency to your app ).

To make this clearer, let's look at an example:

DCli depends on the path package and if you are building a CLI app you are also likely to need it.

if you aren't already using the path package then take another look!

If the DCli pubspec.yaml had a dependency such as:

    dependencies:
        path: 1.1.0

and your pubspec.yaml had:

    dependencies:
        path: 1.2.0

We now have a problem.

The dart pub command can't find a version of path, that will work with DCli and your package.

Our problem is resolved by leaving the version range open by using ^1.x.x on both packages.

    dependencies:
        path: ^1.1.0

and your pubspec.yaml had:

    dependencies:
        path: ^1.2.0

The dart pub command is now free to choose at least version 1.2.0 of path and may choose a later version.

Now, both DCli and your app can be happy, happy, happy.

So this all sounds swell.

If you are building a flutter app it's actually idea.

The dart pub command finds the best version, you compile your app, test it and distribute it as a binary app.

This one is not like the others

So this is where the difference lies.

With a Flutter app, the version of the path package is fixed at compile time and you test against that fixed version.

With a CLI app we are however deploying via source code, so the version of path that is used won't be set until until your colleague runs dart pub global activate mycliapp on their machine.

Just who is in charge here?

So we know understand that you aren't in control of the final version of path that is used but why is that a problem?

The problem is that semantic versioning isn't water tight.

The Dart package ecosystem uses semantic versioning. Each of the three digits in a package version such as '1.0.0' are meant to tell you something about the packages API.

The first digit is the major version no.

A change in the major version number is intended to indicate a break in the API.

If the first digit in a package changes, you can reasonably expect that you code will no longer work with the package without having to make some code changes. This is of course not always true, as even a small breaking change to an API should result in the major version no. being incremented, but if you don't use that particular API then your code should still work.

The Dart version ranges are designed to prevent such breakages.

If you pubspec.yaml contains:

dependencies:
  path: ^1.0.0

Then the dart pub command will only use a version of path that starts with a 1. e.g. 1.x.x

So this is where the problems begin. API breakages are sometimes very subtle and even bug fixes can break your code.

Sometimes bugs are intended!

The WINE package is famous for intentionally replicating bugs in the Windows API to guarantee that programs that rely on those Windows bugs still work.

The result of the lived experience is that, semantic versioning doesn't work for CLI apps distributed by a Dart repository.

Even with the best intent, developers make mistakes with semantic versioning and even when they get it right, minor changes can break your code.

So how do we fix this problem?

A hero is born

Rolling music, swells from the darkness
A distant horizon, a lonely figure in sight
A hat on his head, a chevron on his toe
A key in his pocket, a lock ready to throw
The users are restless, fear billows through the flight
A crash echos, panic takes hold
The lock is closed, the users gasp
A hat is thrown, a crisis averted, 
The herd moves on, the drones appeased

I'm still not certain what that is all about...

The DCli lock command is our de-hatted hero.

The latest version of DCli includes a 'lock' command. Even if you are not using the DCli libraries you will want to use the dcli lock command.

The purpose of the lock command is simple. It locks the version of every dependency to a single version.

The lock command is very similar in purpose to that of the pubspec.lock file.

The pubspec.lock file holds the actual version of each dependency on which your project depends. The pubspec.lock file is created the first time you run 'dart pub get' and will not change unless you run dart pub upgrade

The problem is that the pubspec.lock file doesn't get published to pub.dev (nor OnePub) so it can't be relied upon to lock our versions.

The dcli lock command takes a different approach in that it updates your pubspec.yaml.

name: mycliapp
version: 0.0.1
description: A simple command-line application created by dcli
environment: 
  sdk: '>=2.12.1 <3.0.0'
dependencies: 
  args: ^1.0.0
  dcli: ^1.35.0
  path: ^1.0.0
dev_dependencies: 
  lint_hard: ^1.0.0

After running the dcli lock command you pubspec.yaml might look like:

name: mycliapp
version: 0.0.1
description: A simple command-line application created by dcli
environment: 
  sdk: '>=2.12.1 <3.0.0'
dependencies: 
  _fe_analyzer_shared: 26.0.0
  analyzer: 2.3.0
  archive: 3.2.2
  args: 2.3.1
  async: 2.8.2
  basic_utils: 3.8.0
  boolean_selector: 2.1.0
  charcode: 1.3.1
  chunked_stream: 1.4.1
  circular_buffer: 0.11.0
  cli_util: 0.3.5
  clock: 1.1.1
  collection: 1.16.0
  convert: 3.0.1
  coverage: 1.0.3
  crypto: 3.0.1
  csv: 5.0.1
  dart_console2: 2.0.0
  dcli: 1.35.3
  dcli_core: 1.35.3
  equatable: 2.0.3
  ffi: 2.0.1
  file: 6.1.2
  file_utils: 1.0.1
  frontend_server_client: 2.1.2
  functional_data: 1.0.0
  glob: 2.0.1
  globbing: 1.0.0
  http: 0.13.4
  http_multi_server: 3.2.0
  http_parser: 4.0.0
  ini: 2.1.0
  intl: 0.17.0
  io: 1.0.3
  js: 0.6.3
  json2yaml: 3.0.0
  json_annotation: 4.1.0
  logging: 1.0.2
  matcher: 0.12.11
  meta: 1.7.0
  mime: 1.0.1
  node_preamble: 2.0.1
  package_config: 2.0.2
  path: 1.8.1

  ...

dev_dependencies: 
  lint_hard: ^1.0.0

For brevity I've truncated some of the contents of the pubspec.yaml.

There are two things to note for the update pubspec.yaml.

1) ever version is now a fixed version (the ^ hats are gone) 2) every direct and transitive dependency is included

The transitive dependencies are important as they too can be updated and break your code.

Now we have locked our pubspec.yaml we are ready to publish our CLI app.

dart pub publish

We publish it in the normal manner (after re-running our unit test!).

We can now be guaranteed that our CLI app will work on any target machine even in the face of a changing set of available dependency versions.

A word of caution.

If you CLI app ships as both a CLI application and as a library then using dcli 'lock' is a problem as it will also restrict the libraries dependencies.

In this case you might be better off splitting the package into two separate packages.

Happy coding!

Additional Reading: