dev.hel.fi

Here in Open Software Group at City of Helsinki we have been tracking dependency updates of Python projects with requires.io for some time. But recently we decided to cover all of our projects, Node.js included. I set out to compare all the available services, and hopefully find a single solution that would:

• handle both Python and Node repositories
• automatically tracks all dependencies (even without new commits) and send email and slack notifications for updates when new versions or CVEs are released
• make pull requests with updated dependencies
• check branches and pull requests for outdated and insecure dependencies

To cut the chase, I couldn't find anything that would cover all of those needs. For now we have to settle for two services handling Python and Node each: requires.io and snyk.io.

Snyk.io

Just two weeks ago Snyk announced expanded support for Python. Before they had only CLI tools available for development and integration into CI systems, but that wouldn't have helped us to monitor older projects which are touched only rarely.
I had decided to use Snyk for Node projects anyway, so I was excited about the possibility to move all of our projects there.

Unfortunately after some problems with a few repositories and contacting the Snyk support, I learned that they actually install
the dependencies… with Python2.7. So any dependencies that require Python3, or don't list their subdependencies correctly,
or require something that can't be installed with pip (e.g. C headers or libraries for wrappers) simply fail to give any results.
The otherwise pretty nice dashboard won't help you with this either, so I wasted some time clicking retry and removing and adding the repositories again.

But for Node Snyk has been great. It also handles subdependencies, which is something not all services can do.
Compare how Snyk and Gemnasium report the same project:





Snyk also has a dashboard that shows the status for all enabled repositories, CLI tools, Slack, does PRs (initial PR requires a manual click) and will not list repositories which it doesn't understand (i.e. languages or dependency systems it doesn't know about).

One thing to note is that Snyk only tracks security problems. It won't tell you that there simply is a newer version available.
In general this is good, but does leave a potential problem when newer versions silently (or even accidentally) fix vulnerabilities.

Requires.io

We were already using Requires for some projects. It does the job, but is somewhat unwieldy:

• Docs are very lacking; they only talk about badges and only the frontpage tells you that they can do PRs, emails etc.
• There's no dashboard showing status of all repos
• PRs don't have any information, just the commits

But after looking for alternatives, it still does the crucial things the best. There are also some nifty features,
like setting global notification settings with globbing and setting filters into requirements.txt (for example to prevent updates from LTS to non-LTS versions, or to avoid known bad versions).

pyup.io

Python only, it is in many ways nicer than Requires: it has CLI tools, pretty good dashboard, changelogs in PRs and
can be configured to update all or just insecure dependencies.

But unfortunately it relies fully on PRs and GitHub notifications. But we would like to have separate notifications so we can give them higher priority than usual PRs, or simply have notifications without PRs at all.

Gemnasium

In principle Gemnasium handles Python and Node, but doesn't understand any other files than requirements.txt for Python, such as dev-requirements.txt… It also does PRs only for Ruby projects, even if they have a blog post
talking about other languages from 2014. Finally, Gemnasium didn't notice insecure sub- (deep) dependencies.

A shame, really, since their dashboard was the nicest. Their team was pretty responsive and I'll be checking how they have
progressed when the year changes.

nodesecurity.io

Doesn't have any docs, just the frontpage, and doesn't seem to have GitHub integration and just asks for email to create an account. Didn't seem worth it to look deeper.

david-dm

Has just badges, no notifications or pull requests.

Versioneye

Has a maximum of four free OSS repositories. Also says that their software is OSS, but there are no installation instructions anywhere.

Touchpine

Doesn't seem to handle dependenceis but rather track end user software (to get collated notifications for Apache, tomcat etc.).

Libraries.io

Didn't ask for GitHub organisation permissions, so only saw my personal repositories.

Greenkeeper

Only Node, and only does pull requests like pyup.

sourceclear.com

No GitHub integration and free tier does only ten scans a month.

Doppins.com

Should handle both Node and Pyhton, but doesn't do notifications.

Larger code analysers

There's also a bunch of services that don't specialise in dependency tracking (or don't do them at all) but are rather like advanced linters. While they are out of scope, I checked the most prominent ones (mentioned in StackOverflow and blog posts that talk about dependency trackers).

• Squale hasn't been updated since 2012.
• Panopticode hasn't been updated since 2014 and domain is dead.
• Synopsys, Metrixware, CAST and Veracode don't have free plans for OSS projects, and in fact don't list any prices publicly
• SonarQube / SonarCloud, Coverity, landscape.io, scrutinizer-ci.com, codeclimate and bithound seem promising.
• Bunch of other tools that can be run from CLI only (though possibly communicating with a server somewhere).