In other news, here is some recent progress:‘new’ now requires ‘this’ to be a row type
I changed inference of ‘new’ calls, so that the constructed function must have a row type as it’s “this” implicit parameter (nothing else makes sense).
I added an initial implementation of what I call ambiguous types. These are simple type constraints that limit a type variable to a set of specific types.
With the new type constraints, + has the following type:a = (TNumber | TString) => ((a, a) -> a)
The syntax is reminiscent of Haskell’s type classes, and means: given a type variable “a” that is either a TNumber or a TString, the type of + is: (a, a) -> a
I’m thinking about implementing full-blown type classes, or alternatively, a more powerful form of ambiguous types, to deal with some other more complicated examples.
A minor, barely noticed consultation is not the way to make a major change to Scottish citizens’ privacy and their relationship with the state. Creating a national ID register was rejected by the SNP and the UK, and the bare minimum should be for the Scottish Executive to introduce primary legislation whereby the public and MSPs can debate the nature of these changes and whether they are acceptable.
Respond to the consultation quickly, courtesy of ORG.
ORG is planning meetings to discuss how we can stop the Scottish Government's plans in Edinburgh, Glasgow and Aberdeen, and is tracking developments in their blog.
Here is the original consultation, and a detailed response by ORG.
As we approach the release of GHC 7.10, there is a new wave of Haskell packages that require trivial fixes to build with the new versions of the compiler and standard libraries, but whose authors/maintainers are not around to apply the fixes. This is especially annoying when there is a pull request on GitHub, and all the maintainer would have to do is to press the green Merge button, and upload the new version on hackage.
If you are a responsible maintainer and don’t want this to happen to your packages in the future, you should appoint backup maintainers for your packages.
But what if you are a user of a package that remains broken on hackage, even though a fix is available? Here I review several ways to deal with this problem, including the new and promising Stackage snapshots.Building the package locally
If all you care about is to get something installed locally (be it the broken package itself, or something that directly or indirectly depends on it), you can install the fixed version locally.Non-sandboxed way
Check out the repository or branch with the fix, and cabal-install it:% git clone -b ghc710 https://github.com/markus1189/feed.git % cabal install ./feed
(I prepend ./ to make sure cabal understands that I mean the directory, and not the package name on hackage.)Sandboxed way
If you’re installing in the sandbox, then you can use add-source (although the non-sandboxed version will work in this case, too):% git clone -b ghc710 https://github.com/markus1189/feed.git % cabal sandbox add-source feed % cabal install whatever-you-needed
If the package whatever-you-needed has feed among its transitive dependencies, cabal will automatically install it from the add-source’d directory.Limitations
This approach doesn’t work well if:
You are a maintainer of a package that depends on the broken library. It’s hard to ask your users to check out and build the fixed version by hand.
You work on an in-house application that your coworkers should be able to build, for the same reason.
You cannot upload the fixed version of a package on hackage bypassing the maintainer. However, you can upload it under a new name. This works well if you are a maintainer of a package that directly depends on the broken package, because you can easily make your package depend on your fork.
Examples of this are tasty depending on regex-tdfa-rc (a fork of regex-tdfa) and tasty-golden depending on temporary-rc (a fork of temporary).Limitations
This doesn’t work well if there’s a chain of dependencies leading from your package to the broken one. You have to either persuade the other maintainer(s) to depend on your fork or fork the entire chain.
If the broken package becomes actively developed again, you need to either move back to using it or backport the bugfixes from it to your fork. (I only fork packages when I find this scenario unlikely.)
Other packages that depend on the broken package won’t automatically get fixed.
Some people get upset when you fork packages.
Instead of uploading the fixed version to hackage (which you can’t), you can upload it to Stackage instead, by creating a custom snapshot.
The procedure is described in Experimental package releases via Stackage Server. You create four files:
- The fixed tarball (produced by cabal sdist). You probably want to bump the package’s version, so that it doesn’t conflict with the version already on hackage.
- Two text files: desc and slug. The first one contains a human-readable description of the snapshot; the second contains an id that will become part of the snapshot’s URL.
- A text file with the packages to be copied directly from hackage. For the purpose of this article, you probably want to leave this file empty. (I don’t know if it’s ok not to include it at all.)
Then you pack these four files into a tarball (that’s right, it’ll be a tarball with a tarball inside) and upload to stackage (after registering, if you haven’t registered before).
The outcome will be a custom hackage-like repository which will contain the single version of a single package — the one you’ve uploaded. (Of course, you can include multiple packages or versions if you like.)
The Stackage website will give you the remote-repo line that you can put into your cabal.config along with the hackage or stackage repos that are already there.
In contrast to building packages locally, you can easily tell your users or coworkers to add that repo as well.Limitations
If the new hackage release of the broken package will get the same version number as your stackage version, there will be a conflict. (I actually don’t know what happens in that case; my guess is that cabal will silently pick one of the two available versions.)
If the package you maintain (which depends on the broken package) is a small one, or is deep down the dependency chain, it may be hard to tell your users to add the repository. If, on the other hand, you maintain a major web framework or other such thing, it would probably work.
There’s a procedure for taking over a package described on the wiki. You’ll need to contact the current maintainer; wait an indefinite amount of time (there’s no consensus about it; estimates vary from 2 weeks to 6-12 months); ask again on the mailing list and wait again; finally ask Hackage admins to grant you the rights.Limitations
Since this procedure takes a long time, it’s almost never sufficient by itself, and you’ll need to resort to one of the other strategies until you’re given the upload rights.
It’s not clear how long you actually need to wait.
I find it odd that you need to jump through all these hoops in order to do a service to the community.
I started working on Haskell projects (I am a product manager working for FP Complete) with developers and a pattern is emerging which is there isn't enough tutorials and samples teaching how to build an end-to-end Haskell application!
If you have an application that you developed in Haskell, I would love to hear from you and we can together make it into a tutorial to teach others and share the knowledge promoting and encouraging more developers to use Haskell for commercial applications.
Got an app? Let me know!submitted by saldaoud
[link] [29 comments]
Some of the work I lead at Galois was highlighted in the initial story on 60 Minutes last night, a spot interviewing Dan Kaufman at DARPA. I’m Galois’ principal investigator for the HACMS program, focused on building more reliable software for automobiles and aircraft and other embedded systems. The piece provides a nice overview for the general public on why software security matters and what DARPA is doing about it; HACMS is one piece of that story.
I was busy getting married when filming was scheduled, but two of my colleagues (Dylan McNamee and Pat Hickey) appear in brief cameos in the segment (don’t blink!). Good work, folks! I’m proud of my team and the work we’ve accomplished so far.
You can see more details about how we have been building better programming languages for embedded systems and using them to build unpiloted air vehicle software here.
A few weeks ago I received a bug report against streaming-commons. Since then, the details of what we discovered when discussing this report have been bothering me quite a bit, as they expose a lot of the brittleness of the Haskell toolchain. I'm documenting all of these aspects now to make clear how fragile our tooling is, and thereby explain why I think Stackage is so vital to our community.
In this blog post, I'm going to describe six separate problems I've identified when looking into this issue, and explain how Stackage (or some similar deterministic build system) would have protected users against these problems had it been employed.The story
streaming-commons is a library that provides helper utilities for a number of different streaming concepts, one of them being a streaming way to convert blaze-builder Builders to filled ByteString buffers. Since blaze-builder was released a few years ago, a new set of modules was added to the bytestring package in version 0.10 known as a "bytestring builder." I asked one of the engineers at FP Complete, Emanuel Borsboom, to start working on a new module for streaming-commons to provide similar functionality for bytestring builder.
And now we run into the first problem with the Haskell toolchain. You would think that we should just add a lower bound on bytestring >= 0.10 in the streaming-commons.cabal file. However, setting restrictive lower bounds on ghc-package dependencies can be a problem. Fortunately, Leon Smith already solved this problem for us with bytestring-builder, which provides a compatibility layer for older bytestrings (much like Ed's transformers-compat). The idea is that, when compiled against an older version of bytestring, the bytestring-builder package provides the necessary missing modules, and otherwise does nothing.
When Emanuel wrote his changes to streaming-commons, he added a dependency on bytestring-builder. We then proceeded to test this on multiple versions of GHC via Travis CI and Herbert's multi-ghc-travis. Everything compiled and passed tests, so we shipped the updated version.
However, that original bug report I linked to- reported by Ozgun Ataman- told us there was a problem with GHC 7.6. This was pretty surprising, given that we'd tested on GHC 7.6. Fortunately Lane Seppala discovered the culprit: the Cabal library. It turns out that installing a new version of the Cabal library causes the build of streaming-commons to break, whereas our tests just used the default version of Cabal shipped with GHC 7.6. (We'll get back to why that broke things in a bit.)
After some digging, Emanuel discovered the deeper cause of the problem: Bryan O'Sullivan reported an issue a year ago where- when using a new version of the Cabal library- bytestring-builder does not in fact provide it's compatibility modules. This leads us to our second issue: this known bug existed for almost a year without resolution, and since it only occurs in unusual circumstances, was not detected by any of our automated tooling.
The reason this bug existed though is by far the most worrisome thing I saw in this process: the Cabal library silently changed the semantics of one of its fields in the 1.18 (or 1.20? I'm not sure) release. You see, bytestring-builder was detecting which version of bytestring it was compiled against by inspecting the configConstraints field (you can see the code yourself on Hackage). And starting in Cabal 0.19.1 (a development release), that field was no longer being populated. As a result, as soon as that newer Cabal library was installed, the bytestring-builder package became worse than useless.
As an aside, this points to another problematic aspect of our toolchain: there is no way to specify constraints on dependencies used in custom Setup.hs files. That's actually causes more difficulty than it may sound like, but I'll skip diving into it for now.
The fix for this was relatively simple: use some flag logic in the cabal file instead of a complicated custom Setup.hs file. (Once this pull request was merged in and released, it did fix the original bug report.) But don't take this as a critique of Leon's choice of a complicated Setup.hs file. Because in reality, the flag trick- while the "standard" solution to this problem- broke cabal-install's dependency solver for quite a while. To be fair, I'm still not completely convinced that the bug is fixed, but for now that bug is the lesser of two evils vs the Cabal library bug.
And finally, based on the bug report from Ozgun, it seems like an internal build failed based on all of this occurring. This has been a constant criticism I've made about the way we generally do builds in the Haskell world. Rarely is reproducibility a part of the toolchain. To quote Ozgun:
We are in fact quite careful in dependency management with lower-upper bounds on most outside packages, so breakages like this are unexpected.
And many people feel that this is the way things should be. But as this discussion hopefully emphasizes, just playing with lower and upper bounds is not sufficient to avoid build failures in general. In this case, we're looking at a piece of software that was broken by a change in a library that it didn't depend on, namely Cabal, since our tooling makes an implicit dependency on that library, and we have no way of placing bounds on it.The case for Stackage
So here are the toolchain problems I've identified above:
- Tight coupling between GHC version and some core libraries like bytestring.
- A known issue lasting undocumented for a corner case for over a year, without any indication on the Hackage page that we should be concerned.
- The Cabal library silently changed the semantics of a field, causing complete breakage of a package.
- cabal-install's solver gets confused by standard flag usage, at least in slightly older versions.
- Not all dependencies are actually specified in a cabal file. At the very least, the Cabal library version is unconstrained, and any other packages used by Setup.hs.
- Default Haskell toolchain doesn't protect us against these kinds of problems, or give us any concept of reproducibility.
Stackage completely solves (2), (3), (5), and (6) for end users. By specifying all library versions used, and then testing all of those versions together, we avoid many possible corner cases of weird library interactions, and provide a fully reproducible build. (Note the Stackage doesn't solve all such cases: operating system, system libraries, executables, etc are still unpinned. That's why FP Complete is working on Docker-based tooling.)
(1) is highly mitigated by Stackage because, even though the tight coupling still exists, Stackage provides a set of packages that take that coupling into account for you, so you're not stuck trying to put the pieces together yourself.
As for (4)... Stackage helps the situation by making the job of the solver simpler by pinning down version numbers. Unfortunately, there are still potential gotchas when encountering solver bugs. Sometimes we end up needing to implement terribly awkward solutions to work around those bugs.
Hello, I am trying to learn haskell and try to understand the behavior of haskell programs. So I run following code in ghci Prelude> product [1..1000000]
I was expecting this code to use O(1) space. But when I run the code, I saw that ghc is using 250MB of ram to calculate this. Which suggests that space for the whole list is initialized in memory.
I believe this is the definition of product https://github.com/ghc/packages-base/blob/52c0b09036c36f1ed928663abb2f295fd36a88bb/Data/List.hs#L1010-L1021
In the definition, Comment explains that product function can calculate infinite lists. So there should be something that triggers full list initialization that I am not aware of.
I was just wondering what is the reason of that behavior. And how can I do this calculation in constant space.
Also I am using htop to see memory consumption of ghc. Is there a better way of doing this?
Thankssubmitted by yilmazhuseyin
[link] [20 comments]