OCaml Changelog

This release also contains contributions from: Rudi Grinberg, Fourchaux, Christopher Reichert, David Allsopp, Nick Borden, Mario Rodas, @Twinside, Pierre Chambart, Philipp Haselwarter, Tomasz Kołodziejski and Syohei Yoshida.

OPAM 1.2.2 has just been released. This fixes a few issues over 1.2.1 and brings a couple of improvements, in particular better use of the solver to keep the installation as up-to-date as possible even when the latest version of a package can not be installed.

Upgrade from 1.2.1 (or earlier)

See the normal installation instructions: you should generally pick up the packages from the same origin as you did for the last version -- possibly switching from the official repository packages to the ones we provide for your distribution, in case the former are lagging behind.

There are no changes in repository format, and you can roll back to earlier versions in the 1.2 branch if needed.

Improvements

• Conflict messages now report the original version constraints without translation, and they have been made more concise in some cases
• Some new opam lint checks, opam lint now numbers its warnings and may provide script-friendly output
• Feature to automatically install plugins, e.g. opam depext will prompt to install depext if available and not already installed
• Priority to newer versions even when the latest can't be installed (with a recent solver only. Before, all non-latest versions were equivalent to the solver)
• Added opam list --resolve to list a consistent installation scenario
• Be cool by default on errors in OPAM files, these don't concern end-users and packagers and CI now have opam lint to check them.

Fixes

• OSX: state cache got broken in 1.2.1, which could induce longer startup times. This is now fixed
• opam config report has been fixed to report the external solver properly
• --dry-run --verbose properly outputs all commands that would be run again
• Providing a simple path to an aspcud executable as external solver (through options or environment) works again, for backwards-compatibility
• Fixed a fd leak on solver calls (thanks Ivan Gotovchits)
• opam list now returns 0 when no packages match but no pattern was supplied, which is more helpful in scripts relying on it to check dependencies.

OPAM 1.2.1 has just been released. This patch version brings a number of fixes and improvements over 1.2.0, without breaking compatibility.

Upgrade from 1.2.0 (or earlier)

See the normal installation instructions: you should generally pick up the packages from the same origin as you did for the last version -- possibly switching from the official repository packages to the ones we provide for your distribution, in case the former are lagging behind.

What's new

No huge new features in this point release -- which means you can roll back to 1.2.0 in case of problems -- but lots going on under the hood, and quite a few visible changes nonetheless:

• The engine that processes package builds and other commands in parallel has been rewritten. You'll notice the cool new display but it's also much more reliable and efficient. Make sure to set jobs: to a value greater than 1 in ~/.opam/config in case you updated from an older version.
• The install/upgrade/downgrade/remove/reinstall actions are also processed in a better way: the consequences of a failed actions are minimised, when it used to abort the full command.
• When using version control to pin a package to a local directory without specifying a branch, only the tracked files are used by OPAM, but their changes don't need to be checked in. This was found to be the most convenient compromise.
• Sources used for several OPAM packages may use <name>.opam files for package pinning. URLs of the form git+ssh:// or hg+https:// are now allowed.
• opam lint has been vastly improved.

... and much more

There is also a new manual documenting the file and repository formats.

Fixes

See the changelog for a summary or closed issues in the bug-tracker for an overview.

Experimental features

These are mostly improvements to the file formats. You are welcome to use them, but they won't be accepted into the official repository until the next release.

• New field features: in opam files, to help with ./configure scripts and documenting the specific features enabled in a given build. See the original proposal and the section in the new manual
• The "filter" language in opam files is now well defined, and documented in the manual. In particular, undefined variables are consistently handled, as well as conversions between string and boolean values, with new syntax for converting bools to strings.
• New package flag "verbose" in opam files, that outputs the package's build script to stdout
• New field libexec: in <name>.install files, to install into the package's lib dir with the execution bit set.
• Compilers can now be defined without source nor build instructions, and the base packages defined in the packages: field are now resolved and then locked. In practice, this means that repository maintainers can move the compiler itself to a package, giving a lot more flexibility.

Main new feature is a faster short-path, and also a lot of buxfixes.

This release also contains contributions from: Geoff Gole, Rudi Grinberg, Markus Mottl, Roman Vorobets and Arthur Wendling.

We are very proud to announce the availability of OPAM 1.2.0.

Upgrade from 1.1

Simply follow the usual instructions, using your preferred method (package from your distribution, binary, source, etc.) as documented on the homepage.

NOTE: There are small changes to the internal repository format (~/.opam). It will be transparently updated on first run, but in case you might want to go back and have anything precious there, you're advised to back it up.

Usability

Lot of work has been put into providing a cleaner interface, with helpful behaviour and messages in case of errors.

The documentation pages also have been largely rewritten for consistency and clarity.

New features

This is just the top of the list:

• A extended and versatile opam pin command. See the Simplified packaging workflow
• More expressive queries, see for example opam source
• New metadata fields, including source repositories, bug-trackers, and finer control of package behaviour
• An opam lint command to check the quality of packages

For more detail, see the announcement for the beta, the full changelog, and the bug-tracker.

Package format

The package format has been extended to the benefit of both packagers and users. The repository already accepts packages in the 1.2 format, and this won't affect 1.1 users as a rewrite is done on the server for compatibility with 1.1.

If you are hosting a repository, you may be interested in these administration scripts to quickly take advantage of the new features or retain compatibility.

After a few months of development, we are pleased to announce the stable release of Merlin 2.0.
Supported OCaml versions range from 4.00.1 to 4.02.1.

Overview

Merlin is a tool focused on helping you code in OCaml by providing features such as:

• automatic completion of identifiers, using scope and type information,
• interactively typing definitions and expressions during edition,
• jumping to the definition of any identifier,
• quickly reporting errors in the editor.

We provide integration into Vim and Emacs. An external plugin is also available for Sublime Text.

What's new

This release provides great improvements in robustness and quality of analysis. Files that changed on disk are now automatically reloaded. The parsing process is finer grained to provide more accurate recovery and error messages. Integration with Jane Street Core and js_of_ocaml has also improved.

Vim & Emacs are still the main targeted editors. Thanks to Luc Rocher, preliminary support for Sublime Text is also available, see Sublime-text-merlin. Help is welcome to improve and extend supported editing environments.

Windows support also received some fixes. Merlin is now distributed in WODI. Integration in OCaml-on-windows is planned.

Installation

This new version of Merlin is already available with opam using opam install merlin, and can also be built from the sources which are available at the-lambda-church/merlin.

Minor update to installation procedure

Most package managers support some pin functionality to ensure that a given package remains at a particular version without being upgraded. The stable OPAM 1.1 already supported this by allowing any existing package to be pinned to a target, which could be a specific released version, a local filesystem path, or a remote version-controlled repository.

However, the OPAM 1.1 pinning workflow only lets you pin packages that already exist in your OPAM repositories. To declare a new package, you had to go through creating a local repository, registering it in OPAM, and adding your package definition there. That workflow, while reasonably clear, required the user to know about the repository format and the configuration of an internal repository in OPAM before actually getting to writing a package. Besides, you were on your own for writing the package definition, and the edit-test loop wasn't as friendly as it could have been.

A natural, simpler workflow emerged from allowing users to pin new package names that don't yet exist in an OPAM repository:

1. choose a name for your new package
2. opam pin add in the development source tree
3. the package is created on-the-fly and registered locally.

To make it even easier, OPAM can now interactively help you write the package definition, and you can test your updates with a single command. This blog post explains this new OPAM 1.2 functionality in more detail; you may also want to check out the new Packaging tutorial relying on this workflow.

From source to package

For illustration purposes in this post I'll use a tiny tool that I wrote some time ago and never released: ocp-reloc. It's a simple binary that fixes up the headers of OCaml bytecode files to make them relocatable, which I'd like to release into the public OPAM repository.

"opam pin add"

The command opam pin add <name> <target> pins package <name> to <target>. We're interested in pinning the ocp-reloc package name to the project's source directory.

cd ocp-reloc
opam pin add ocp-reloc .

If ocp-reloc were an existing package, the metadata would be fetched from the package description in the OPAM repositories. Since the package doesn't yet exist, OPAM 1.2 will instead prompt for on-the-fly creation:

Package ocp-reloc does not exist, create as a NEW package ? [Y/n] y
ocp-reloc is now path-pinned to ~/src/ocp-reloc

NOTE: if you are using beta4, you may get a version-control-pin instead, because we added auto-detection of version-controlled repos. This turned out to be confusing (issue #1582), because your changes wouldn't be reflected until you commit, so this has been reverted in favor of a warning. Add the --kind path option to make sure that you get a path-pin.

OPAM Package Template

Now your package still needs some kind of definition for OPAM to acknowledge it; that's where templates kick in, the above triggering an editor with a pre-filled opam file that you just have to complete. This not only saves time in looking up the documentation, it also helps getting consistent package definitions, reduces errors, and promotes filling in optional but recommended fields (homepage, etc.).

opam-version: "1.2"
name: "ocp-reloc"
version: "0.1"
maintainer: "Louis Gesbert <louis.gesbert@ocamlpro.com>"
authors: "Louis Gesbert <louis.gesbert@ocamlpro.com>"
homepage: ""
bug-reports: ""
license: ""
build: [
  ["./configure" "--prefix=%{prefix}%"]
  [make]
]
install: [make "install"]
remove: ["ocamlfind" "remove" "ocp-reloc"]
depends: "ocamlfind" {build}

After adding some details (most importantly the dependencies and build instructions), I can just save and exit. Much like other system tools such as visudo, it checks for syntax errors immediately:

[ERROR] File "/home/lg/.opam/4.01.0/overlay/ocp-reloc/opam", line 13, character 35-36: '.' is not a valid token.
Errors in /home/lg/.opam/4.01.0/overlay/ocp-reloc/opam, retry editing ? [Y/n]

Installation

You probably want to try your brand new package right away, so OPAM's default action is to try and install it (unless you specified -n):

ocp-reloc needs to be installed.
The following actions will be performed:
 - install   cmdliner.0.9.5                        [required by ocp-reloc]
 - install   ocp-reloc.0.1*
=== 1 to install ===
Do you want to continue ? [Y/n]

I usually don't get it working the first time around, but opam pin edit ocp-reloc and opam install ocp-reloc -v can be used to edit and retry until it does.

Package Updates

How do you keep working on your project as you edit the source code, now that you are installing through OPAM? This is as simple as:

opam upgrade ocp-reloc

This will pick up changes from your source repository and reinstall any packages that are dependent on ocp-reloc as well, if any.

So far, we've been dealing with the metadata locally used by your OPAM installation, but you'll probably want to share this among developers of your project even if you're not releasing anything yet. OPAM takes care of this by prompting you to save the opam file back to your source tree, where you can commit it directly into your code repository.

cd ocp-reloc
git add opam
git commit -m 'Add OPAM metadata'
git push

Publishing your New Package

The above information is sufficient to use OPAM locally to integrate new code into an OPAM installation. Let's look at how other developers can share this metadata.

Picking up your development package

If another developer wants to pick up ocp-reloc, they can directly use your existing metadata by cloning a copy of your repository and issuing their own pin.

git clone git://github.com/OCamlPro/ocp-reloc.git
opam pin add ocp-reloc/

Even specifying the package name is optional since this is documented in ocp-reloc/opam. They can start hacking, and if needed use opam pin edit to amend the opam file too. No need for a repository, no need to share anything more than a versioned opam file within your project.

Cloning already existing packages

We have been focusing on an unreleased package, but the same functionality is also of great help in handling existing packages, whether you need to quickly hack into them or are just curious. Let's consider how to modify the omd Markdown library.

opam source omd --pin
cd omd.0.9.7
...patch...
opam upgrade omd

The new opam source command will clone the source code of the library you specify, and the --pin option will also pin it locally to ensure it is used in preference to all other versions. This will also take care of recompiling any installed packages that are dependent on omd using your patched version so that you notice any issues right away.

There's a new OPAM field available in 1.2 called dev-repo. If you specify this in your metadata, you can directly pin to the upstream repository via opam source --dev-repo --pin.

If the upstream repository for the package contains an opam file, that file will be picked up in preference to the one from the OPAM repository as soon as you pin the package. The idea is to have:

• a development opam file that is versioned along with your source code (and thus accurately tracks the latest dependencies for your package).
• a release opam file that is published on the OPAM repository and can be updated independently without making a new release of the source code.

How to get from the former to the latter will be the subject of another post! In the meantime, all users of the beta are welcome to share their experience and thoughts on the new workflow on the bug tracker.

It has only been 18 months since the first release of OPAM, but it is already difficult to remember a time when we did OCaml development without it. OPAM has helped bring together much of the open-source code in the OCaml community under a single umbrella, making it easier to discover, depend on, and maintain OCaml applications and libraries. We have seen steady growth in the number of new packages, updates to existing code, and a diverse group of contributors.

OPAM has turned out to be more than just another package manager. It is also increasingly central to the demanding workflow of industrial OCaml development, since it supports multiple simultaneous (patched) compiler installations, sophisticated package version constraints that ensure statically-typed code can be recompiled without conflict, and a distributed workflow that integrates seamlessly with Git, Mercurial or Darcs version control. OPAM tracks multiple revisions of a single package, thereby letting packages rely on older interfaces if they need to for long-term support. It also supports multiple package repositories, letting users blend the global stable package set with their internal revisions, or building completely isolated package universes for closed-source products.

Since its initial release, we have been learning from the extensive feedback from our users about how they use these features as part of their day-to-day workflows. Larger projects like XenAPI, the Ocsigen web suite, and the Mirage OS publish OPAM remotes that build their particular software suites. Complex applications such as the Pfff static analysis tool and Hack language from Facebook, the Frenetic SDN language and the Arakoon distributed key store have all appeared alongside these libraries. Jane Street pushes regular releases of their production Core/Async suite every couple of weeks.

One pleasant side-effect of the growing package database has been the contribution of tools from the community that make the day-to-day use of OCaml easier. These include the utop interactive toplevel, the IOCaml browser notebook, and the Merlin IDE extension. While these tools are an essential first step, there's still some distance to go to make the OCaml development experience feel fully integrated and polished.

Today, we are kicking off the next phase of evolution of OPAM and starting the journey towards building an OCaml Platform that combines the OCaml compiler toolchain with a coherent workflow for build, documentation, testing and IDE integration. As always with OPAM, this effort has been a collaborative effort, coordinated by the OCaml Labs group in Cambridge and OCamlPro in France. The OCaml Platform builds heavily on OPAM, since it forms the substrate that pulls together the tools and facilitates a consistent development workflow. We've therefore created this blog on opam.ocaml.org to chart its progress, announce major milestones, and eventually become a community repository of all significant activity.

Major points:

• OPAM 1.2 beta available: Firstly, we're announcing the availability of the OPAM 1.2 beta, which includes a number of new features, hundreds of bug fixes, and pretty new colours in the CLI. We really need your feedback to ensure a polished release, so please do read the release notes below.

• In the coming weeks, we will provide an overview of what the OCaml Platform is (and is not), and describe an example workflow that the Platform can enable.

• Feedback: If you have questions or comments as you read these posts, then please do join the platform@lists.ocaml.org and make them known to us.

Releasing the OPAM 1.2 beta4

We are proud to announce the latest beta of OPAM 1.2. It comes packed with new features, stability and usability improvements. Here the highlights.

Binary RPMs and DEBs!

We now have binary packages available for Fedora 19/20, CentOS 6/7, RHEL7, Debian Wheezy and Ubuntu! You can see the full set at the OpenSUSE Builder site and download instructions for your particular platform.

An OPAM binary installation doesn't need OCaml to be installed on the system, so you can initialize a fresh, modern version of OCaml on older systems without needing it to be packaged there. On CentOS 6 for example:

cd /etc/yum.repos.d/
wget http://download.opensuse.org/repositories/home:ocaml/CentOS_6/home:ocaml.repo
yum install opam
opam init --comp=4.01.0

Simpler user workflow

For this version, we focused on improving the user interface and workflow. OPAM is a complex piece of software that needs to handle complex development situations. This implies things might go wrong, which is precisely when good support and error messages are essential. OPAM 1.2 has much improved stability and error handling: fewer errors and more helpful messages plus better state backups when they happen.

In particular, a clear and meaningful explanation is extracted from the solver whenever you are attempting an impossible action (unavailable package, conflicts, etc.):

$ opam install mirage-www=0.3.0
The following dependencies couldn't be met:
  - mirage-www -> cstruct < 0.6.0
  - mirage-www -> mirage-fs >= 0.4.0 -> cstruct >= 0.6.0
Your request can't be satisfied:
  - Conflicting version constraints for cstruct

This sets OPAM ahead of many other package managers in terms of user-friendliness. Since this is made possible using the tools from irill (which are also used for Debian), we hope that this work will find its way into other package managers. The extra analyses in the package solver interface are used to improve the health of the central package repository, via the OPAM Weather service.

And in case stuff does go wrong, we added the opam upgrade --fixup command that will get you back to the closest clean state.

The command-line interface is also more detailed and convenient, polishing and documenting the rough areas. Just run opam <subcommand> --help to see the manual page for the below features.

• More expressive queries based on dependencies.

  $ opam list --depends-on cow --rec
  # Available packages recursively depending on cow.0.10.0 for 4.01.0:
  cowabloga   0.0.7  Simple static blogging support.
  iocaml      0.4.4  A webserver for iocaml-kernel and iocamljs-kernel.
  mirage-www  1.2.0  Mirage website (written in Mirage)
  opam2web    1.3.1 (pinned)  A tool to generate a website from an OPAM repository
  opium       0.9.1  Sinatra like web toolkit based on Async + Cohttp
  stone       0.3.2  Simple static website generator, useful for a portfolio or documentation pages
  

• Check on existing opam files to base new packages from.

  $ opam show cow --raw
  opam-version: "1"
  name: "cow"
  version: "0.10.0"
  [...]
  

• Clone the source code for any OPAM package to modify or browse the interfaces.

  $ opam source cow
  Downloading archive of cow.0.10.0...
  [...]
  $ cd cow.0.10.0
  

We've also improved the general speed of the tool to cope with the much bigger size of the central repository, which will be of importance for people building on low-power ARM machines, and added a mechanism that will let you install newer releases of OPAM directly from OPAM if you choose so.

Yet more control for the packagers

Packaging new libraries has been made as straight-forward as possible. Here is a quick overview, you may also want to check the OPAM 1.2 pinning post.

opam pin add <name> <sourcedir>

will generate a new package on the fly by detecting the presence of an opam file within the source repository itself. We'll do a followup post next week with more details of this extended opam pin workflow.

The package description format has also been extended with some new fields:

• bug-reports: and dev-repo: add useful URLs
• install: allows build and install commands to be split,
• flags: is an entry point for several extensions that can affect your package.

Packagers can limit dependencies in scope by adding one of the keywords build, test or doc in front of their constraints:

depends: [
  "ocamlfind" {build & >= 1.4.0}
  "ounit" {test}
]

Here you don't specifically require ocamlfind at runtime, so changing it won't trigger recompilation of your package. ounit is marked as only required for the package's build-test: target, i.e. when installing with opam install -t. This will reduce the amount of (re)compilation required in day-to-day use.

We've also made optional dependencies more consistent by removing version constraints from the depopts: field: their meaning was unclear and confusing. The conflicts field is used to indicate versions of the optional dependencies that are incompatible with your package to remove all ambiguity:

depopts: [ "async" {>= "109.15.00"} & "async_ssl" {>= "111.06.00"} ]

becomes:

depopts: [ "async" "async_ssl" ]
conflicts: [ "async" {< "109.15.00"}
             "async_ssl" {< "111.06.00"} ]

There is an upcoming features field that will give more flexibility in a clearer and consistent way for such complex cases.

Easier to package and install

Efforts were made on the build of OPAM itself as well to make it as easy as possible to compile, bootstrap or install. There is no more dependency on camlp4 (which has been moved out of the core distribution in OCaml 4.02.0), and the build process is more conventional (get the source, run ./configure, make lib-ext to get the few internal dependencies, make and make install). Packagers can use make cold to build OPAM with a locally compiled version of OCaml (useful for platforms where it isn't packaged), and also use make download-ext to store all the external archives within the source tree (for automated builds which forbid external net access).

The whole documentation has been rewritten as well, to be better focused and easier to browse. Please leave any feedback or changes on the documentation on the issue tracker.

Try it out !

The public beta of OPAM 1.2 is just out. You're welcome to give it a try and give us feedback before we roll out the release!

We'd be most interested on feedback on how easily you can work with the new pinning features, on how the new metadata works for you... and on any errors you may trigger that aren't followed by informative messages or clean behaviour.

If you are hosting a repository, the administration scripts may help you quickly update all your packages to benefit from the new features.