When I recently wrote about porting my haskell-testing-stub project to tasty I mentioned that test-framework still has more libraries than tasty. I decided to contribute to changing that and released two small packages that extend tasty with extra functionality:
- tasty-hunit-adapter allows to import existing HUnit tests into tasty (hackage, github): module Main where import Test.HUnit ( (~:), (@=?) ) import Test.Tasty ( defaultMain, testGroup ) import Test.Tasty.HUnit.Adapter ( hUnitTestToTestTree ) main :: IO () main = defaultMain $ testGroup "Migrated from HUnit" $ hUnitTestToTestTree ("HUnit test" ~: 2 + 2 @=? 4)
- tasty-program allows to run external program and test whether it terminates successfully (hackage, github): module Main ( main ) where import Test.Tasty import Test.Tasty.Program main :: IO () main = defaultMain $ testGroup "Compilation with GHC" $ [ testProgram "Foo" "ghc" ["-fforce-recomp", "foo.hs"] Nothing ]
This package has only this basic functionality at the moment. A missing feature is the possibility of logging stdout and stderr to a file so that it can later be inspected or perhaps used by a golden test (but for the latter tasty needs test dependencies).
A few months ago, Petr Pudlák kicked off the conduit-extra package with the addition of ZipSink. As a refresher, ZipSink provides an alternative Applicative instance for Sink which allows multiple Sinks to consume the same stream.
I've just release version 1.0.13 of conduit which promotes ZipSink from conduit-extra into conduit itself. This abstraction has proven to be generally useful, and I hope others enjoy it as well. If you want a more in-depth review of it, please see the original blog post. The only change since then is renaming broadcast to sequenceSinks.
Along with this change, version 1.0.13 adds a new, similar concept: ZipSource and sequenceSources. The idea here is to combine together multiple streams, instead of sequencing one stream after another.
As a simple motivating example, let's say we have some files on the filesystem, where each file contains a list of student test scores. We want to combine together the test scores for all students for each of the tests. The following program does the job:import Control.Monad.Trans.Class (lift) import qualified Data.ByteString.Char8 as S8 import Data.Conduit import qualified Data.Conduit.Binary as CB import qualified Data.Conduit.List as CL import qualified Data.Map as Map type Name = String people :: [Name] people = words "alice bob charlie" files :: Map.Map Name FilePath files = Map.fromList $ map (\name -> (name, name ++ ".txt")) people scores :: MonadResource m => FilePath -> Source m Int scores fp = CB.sourceFile fp $= CB.lines $= CL.map (read . S8.unpack) sources :: MonadResource m => Map.Map Name (Source m Int) sources = fmap scores files sources' :: MonadResource m => Source m (Map.Map Name Int) sources' = sequenceSources sources main :: IO () main = runResourceT $ sources' $$ CL.mapM_ (lift . print)
The important bit is the definition of sources'. We use sequenceSources to combine together each of the individual test scores into a single test score map. With some basic input files, the output looks like:fromList [("alice",1),("bob",3),("charlie",2)] fromList [("alice",2),("bob",2),("charlie",2)] fromList [("alice",3),("bob",1),("charlie",2)]
This is the first blog post by our colleague Austin. Austin is one of our GHC engineers, part of the GHC HQ team and carries the heavy burden of being GHC release manager.
Today, GHC 7.8.1 RC1 has been released after many months of work, with a huge number of new features.
There are some great highlights in the new release, including:
- A host of I/O manager improvements, which results in significantly better scalability on multicore machines (up to 32 to 40 cores).
- A host of new language features: closed type families, pattern synonyms, roles, typed holes, and a basic solver for type-level natural numbers. There's also a completely upgraded version of Template Haskell, now with both typed and untyped flavours.
- The new code generator is now live. There are some new and improved optimizations, like better support for turning self-recursive tail calls into loops. Along with that, there's some preliminary support for SIMD operations in the LLVM backend, and other optimization improvements.
- Improvements to the compiler and build system, including a parallel ghc --make -j, and significantly better (but not yet perfect) cross compilation support. We've also merged full iOS support, so it should be possible to build an iOS compiler with the main tree.
- Dozens of other improvements, new features, and tons of fixed bugs (OS X 10.9 support, dynamic GHCi, Applicative-Monad warnings, etc).
See the release notes for the full details.
You can download the sources or pre-built binaries for your platform.
How big has this release been? The other day I did an approximated diffstat between the 7.6 and 7.8 branches:2784 non-merge changesets 964 files changed, 99952 insertions(+), 116187 deletions(-)
That's just in GHC itself, not including the extra repositories. We've had contributions from over 70 people this release, including 15 new committers in 2013 and 2014. As always, GHC HQ and our many other contributors have yet more stuff planned for the future (Overloaded Record Fields, Applicative Do, further integer-gmp optimizations, and more type system features).
This release includes over a year of hard work by many people, and hopefully it will turn out to be a great release (finally). But please try the RC — it's much easier for us to fix bugs before it's done!
And on that note: we're planning on doing an RC2 soon, as we know there are some current issues for people using platforms like OS X Mavericks, and packages like lens. As we've learned, there's eventually diminishing returns, so we're getting it out now in the hopes we can swat the last few things swiftly with your help.
I’ve been doing a lot of experimenting with concurrent operations in haskell and in particular playing with and thinking about the design of concurrent FIFO queues. These structures are difficult to make both efficient and correct, due to the effects of contention on the parts of the structure tasked with coordinating reads and writes from multiple threads.
These are my thoughts so far on FIFO semantics.FIFO? And how!
In the interesting paper “How FIFO is your concurrent FIFO queue?”(PDF). A Haas, et al. propose that an ideal FIFO queue has operations that are instantaneous (think of each write having an infinitely accurate timestamp, and each read taking the corresponding element in timestamp order). They then measure the degree to which real queues of various designs deviate from this platonic FIFO semantics in their message ordering, using a metric they call “element-fairness”. They experimentally measure element-fairness of both so-called “strict FIFO” as well as “relaxed FIFO” designs, in which elements are read in more or less the order they were written (some providing guarantees of degree of re-ordering, others not).
The first interesting observation they make is that no queue actually exhibits FIFO semantics by their metric; this is because of the realities of the way atomic memory operations like CAS may arbitrarily reorder a set of contentious writes.
The second interesting result is that the efficient-but-relaxed-FIFO queues which avoid contention by making fewer guarantees about message ordering often perform closer to ideal FIFO semantics (by their metric) than the “strict” but slower queues!Observable FIFO Semantics
As an outsider, reading papers on FIFO queue designs I get the impression that what authors mean by “the usual FIFO semantics” is often ill-defined. Clearly they don’t mean the platonic zero-time semantics of the “How FIFO… ” paper, since they can’t be called FIFO by that measure.
I suspect what makes a queue “strict FIFO” (by the paper’s categorization) might simply be
If write x returns at time T, then x will be read before the elements of any writes that have not yet started by time T.
The idea is difficult to express, but is essentially that FIFO semantics is only observable by way of actions taken by a thread after returning from a write (think: thread A writes x, then tells B which writes y, where our program’s correctness depends on the queue returning y after x). Note that since a queue starts empty this is also sufficient to ensure writes don’t “jump ahead” of writes already in the queue.
Imagine an absurd queue whose write never returns; there’s very little one can say for certain about the “correct” FIFO ordering of writes in that case, especially when designing a program with a preempting scheduler that’s meant to be portable. Indeed the correctness criterion above is actually probably a lot stricter than many programs require; e.g. when there is no coordination between writers, an observably-FIFO queue need only ensure that no reader thread sees two messages from the same writer thread out of order (I think).
The platonic zero-time FIFO ordering criterion used in the paper is quite different from this observable, correctness-preserving FIFO criterion; I can imagine it being useful for people designing “realtime” software.Conclusion
At a certain level of abstraction, correct observable FIFO semantics shouldn’t be hard to make efficient; after all, the moments during which we have contention (and horrible performance) are also the moments during which we don’t care about (or have no way of observing) correct ordering. In other words (although we have to be careful of the details) a thread-coordination scheme that breaks down (w/r/t element-fairness) under contention isn’t necessarily a problem. Compare-and-swap does just that, unfortunately it breaks down in a way that is slower rather than faster.