add-lib

This is a work-in-progress / sneak-peek of a new feature in tools.deps.alpha targeted at the problem of dynamically adding libraries while working at the REPL. For example, perhaps you’re working along and discover a need to use custom memoization and want to use core.memoize.

One option is to exit your repl, modify your deps.edn file and restart your repl. But wouldn’t it be nice if you could do that dynamically? There are of course libraries that do dynamic library loading already and some that also give you isolation, but we were looking to give you something more integrated with deps.edn and tools.deps.alpha (so you can also take advantage of git deps, local deps, and whatever future kinds of deps are implemented).

As all the details of this feature are still in flux, we have not yet added it to tools.deps.alpha, but you can use clj and a git dep from the branch to try it out if you like:

$ clj -Sdeps "{:deps \
               {org.clojure/tools.deps.alpha \ 
                {:git/url \"https://github.com/clojure/tools.deps.alpha.git\" \
                 :sha \"d492e97259c013ba401c5238842cd3445839d020\"}}}"
Cloning: https://github.com/clojure/tools.deps.alpha.git
Checking out: https://github.com/clojure/tools.deps.alpha.git 
at d492e97259c013ba401c5238842cd3445839d020
Clojure 1.9.0

user=> (use 'clojure.tools.deps.alpha.repl)
nil

user=> (add-lib 'org.clojure/core.memoize {:mvn/version "0.7.1"})
Downloading: org/clojure/core.memoize/0.7.1/core.memoize-0.7.1.pom
Downloading: org/clojure/core.cache/0.7.1/core.cache-0.7.1.pom
Downloading: org/clojure/data.priority-map/0.0.7/data.priority-map-0.0.7.pom
Downloading: org/clojure/core.memoize/0.7.1/core.memoize-0.7.1.jar
Downloading: org/clojure/core.cache/0.7.1/core.cache-0.7.1.jar
Downloading: org/clojure/data.priority-map/0.0.7/data.priority-map-0.0.7.jar
true
user=> (require 'clojure.core.memoize)
nil

The call to add-lib takes a lib and a coordinate, exactly what you’d find in a deps.edn file and supports all of the current coordinate types (Maven, git, or local). add-lib uses the resolver in tools.deps.alpha to determine the transitive set of dependencies, download them to your Maven cache (if needed) and add the urls of those libs to the highest level DynamicClassLoader in your current thread. add-lib returns true if it added any urls. You can then use namespaces from the dependencies without restarting your REPL.

This technique for expanding the libraries available to the current context will not work in all REPL situations. While some of those constraints are known, I’d be interested to learn more about where this does or doesn’t work. It may be possible that changes in Clojure core could make it more readily accessible.

Additionally, this code is not just blindly adding dependencies you ask for but is actually considering them in the context of the dependencies already on your classpath. So if for example, you already had a version of org.clojure/data.priority-map above, it would do the dependency resolution in the context of that version and not resolve to or add a new version to the classpath.

The tools.deps.alpha.libmap namespace contains code to load the starting lib map and maintain the runtime version as libs and their dependencies are added. The lib map can be loaded by several means:

  • System property clojure.libfile - referring to an edn file holding the lib map. clj will set this automatically for you and point to the cached lib map it’s using.
  • System property clojure.libmap - referring to an edn string holding the lib map data, useful for tools that construct the classpath via means other than clj.
  • System property java.class.path - by analyzing the Java classpath and using standard location information about the Maven local repository and structure, the libmap can be inferred.

How all of this works and which options are available is likely to change. Eventually some of this may even move into Clojure itself in the future.

A great place to chat about this is in #tools-deps in Clojurians slack.

Written on May 4, 2018