News aggregator

FP Complete: Distributing our packages without a sysadmin

Planet Haskell - Tue, 05/12/2015 - 6:00pm

At FP Complete, we're no strangers to running complex web services. But we know from experience that the simplest service to maintain is one someone else is managing for you. A few days ago I described how secure package distribution with stackage-update and stackage-install works, focusing on the client side tooling. Today's blog post is about how we use Amazon S3, Github, and Travis CI to host all of this with (almost) no servers of our own (that caveat explained in the process).

Making executables available

We have two different Haskell tools needed to this hosting: hackage-mirror to copy the raw packages to Hackage, and all-cabal-hashes-tool to populate the raw cabal files with hash/package size information. But we don't want to have to compile this executables every time we call them. Instead, we'd like to simply download and run a precompiled executable.

Like many other Github projects, these two utilize Travis CI to build and test the code every time a commit is pushed. But that's not all; using Travis's deployment capability, they also upload an executable to S3.

Figuring out the details of making this work is a bit tricky, so it's easiest to just look at the .travis.yml file. For the security conscious: the trick is that Travis allows us to encrypt data so that no one but Travis can decrypt it. Then, Travis can decrypt and upload it to S3 for us.

Result: a fully open, transparent process for executable building that can be reviewed by anyone in the community, without allowing private credentials to be leaked. Also, notice how none of our own servers needed to get involved.

Running the executables

We're going to leverage Travis yet again, and use it to run the executables it so politely generated for us. We'll use all-cabal-hashes as our demonstration, though all-cabal-packages works much the same way. We have an script which downloads and runs our executable, and then commits, signs, and pushes to Github. In order to sign and push, however, we need to have a GPG and SSH key, respectively.

Once again, Travis's encryption capabilities come into play. In the .travis.yml file, we decrypt a tar file containing the GPG and SSH key, put them in the correct location, and also configure Git. Then we call out to the script. One wrinkle here is that Travis only supports having a single encrypted file per repo, which is why we have to tar together the two different keys, which is a minor annoyance.

As before, we have processes running on completely open, auditable systems. Uploads are being made to providers we don't manage (either Amazon or Github). The only thing kept hidden are the secrets themselves (keys). And if the process ever fails, I get an immediate notification from Travis. So far, that's only happened when I was playing with the build or Hackage was unresponsive.

Running regularly

It wouldn't be very useful if these processes weren't run regularly. This is a perfect place for a cron job. Unfortunately, Travis doesn't yet support cron job, though they seem to be planning it for the future. In the meanwhile, we do have to run this on our own service. Fortunately, it's a simple job that just asks Travis to restart the last build it ran for each repository.

To simplify even further, I run the Travis command line client from inside a Docker container, so that the only host system dependency is Docker itself. The wrapper script is:

#!/bin/bash set -e set -x docker run --rm -v /home/ubuntu/ bilge/travis-cli /

The script that runs inside the Docker container is the following (token hidden to protect... well, me).

#!/bin/bash set -ex travis login --skip-version-check --org --github-token XXXXXXXXX # Trigger the package mirroring first, since it's used by all-cabal-hashes BUILD=$(travis branches --skip-version-check -r commercialhaskell/all-cabal-packages | grep "^hackage" | awk "{ print \$2 }") BUILDNUM=${BUILD###} echo BUILD=$BUILD echo BUILDNUM=$BUILDNUM travis restart --skip-version-check -r commercialhaskell/all-cabal-packages $BUILDNUM BUILD=$(travis branches --skip-version-check -r commercialhaskell/all-cabal-files | grep "^hackage" | awk "{ print \$2 }") BUILDNUM=${BUILD###} echo BUILD=$BUILD echo BUILDNUM=$BUILDNUM travis restart --skip-version-check -r commercialhaskell/all-cabal-files $BUILDNUM # Put in a bit of a delay to allow the all-cabal-packages job to finish. If # not, no big deal, next job will pick up the change. sleep 30 BUILD=$(travis branches --skip-version-check -r commercialhaskell/all-cabal-hashes | grep "^hackage" | awk "{ print \$2 }") BUILDNUM=${BUILD###} echo BUILD=$BUILD echo BUILDNUM=$BUILDNUM travis restart --skip-version-check -r commercialhaskell/all-cabal-hashes $BUILDNUMConclusion

Letting someone else deal with our file storage, file serving, executable building, and update process is a massive time saver. Now our sysadmins can stop dealing with these problems, and start solving complicated problems. The fact that everyone can inspect, learn from, and understand what our services are doing is another advantage. I encourage others to try out these kinds of deployments whenever possible.

Categories: Offsite Blogs

What do you call it when [[forall x. (x <> x) === x]]?

Haskell on Reddit - Tue, 05/12/2015 - 3:56pm

Is there a name for this specialization of Monoid?

Some Monoids that satisfy this property:

class Monoid a => Prop a where -- only requirement is that (x <> x) === x
  • instance Prop ()
  • instance Prop (First a)
  • instance Prop (Last a)
  • instance Prop Any
  • instance Prop All
  • instance Prop Ordering
  • instance Prop a => Prop (Maybe a)
  • instance Prop a => Prop (Dual a)

Some that don't:

  • Endo
  • Sum
  • Product
  • [a]
submitted by drb226
[link] [6 comments]
Categories: Incoming News

Cloud Haskell success stories?

Haskell on Reddit - Tue, 05/12/2015 - 2:54pm

Has anyone actually used Cloud Haskell successfully in production?

Or is still mostly an academic project?

submitted by jfurmankiewicz
[link] [10 comments]
Categories: Incoming News

Call for participation: Applied FunctionalProgramming (AFP) Summerschool 6-17 July 2015, Utrecht, Netherlands

haskell-cafe - Tue, 05/12/2015 - 10:58am
=========== AFP Summerschool 2015 =========== Applied Functional Programming (AFP) Summerschool July 6-17, 2015 Utrecht University, Department of Information and Computing Sciences Utrecht, The Netherlands Summerschool & registration website: AFP website : contact : Uscs-afp< at > *** The 2015 edition of the Applied Functional Programming (AFP) Summerschool in Utrecht, Netherlands will be held from 6-17 July 2015. The summerschool teaches Haskell on both beginners and advanced levels via lectures and lab exercises. More info can be found via the references above, included here is a summary from the summerschool info: ``Typed functional programming in Haskell allows for the development of compact programs in minimal time and with maximal guarantees about robustness and correctness. The course introduces Haskell as well as its theoretical underpinnings such a
Categories: Offsite Discussion

Haskell: Haskell Sorcery

Haskell on Reddit - Tue, 05/12/2015 - 8:59am
Categories: Incoming News

Proposal: Generalize forever to Applicative

libraries list - Tue, 05/12/2015 - 6:05am
This looks like a no-brainer to me: forever :: Applicative f => f a -> f b forever a = let x = a *> x in x _______________________________________________ Libraries mailing list Libraries< at >
Categories: Offsite Discussion

3rd CFP FARM 2015 -- final reminder

haskell-cafe - Tue, 05/12/2015 - 5:53am
************************************************************ 3rd CFP FARM 2015 3rd ACM SIGPLAN International Workshop on Functional Art, Music, Modelling and Design Vancouver, Canada, 5 September, 2015 affiliated with ICFP 2015 Full Paper and Demo Abstract submission Deadline: 17 May ************************************************************ The ACM SIGPLAN International Workshop on Functional Art, Music, Modelling and Design (FARM) gathers together people who are harnessing functional techniques in the pursuit of creativity and expression. Functional Programming has emerged as a mainstream software development paradigm, and its artistic and creative use is booming. A growing number of software toolkits, frameworks and environments for art, music and design now employ functional programming languages and techniques. FARM is a forum for exploration and critical evaluation of these developments, for example to consider potenti
Categories: Offsite Discussion

trying to go from FFT type to Vector.Storable

haskell-cafe - Tue, 05/12/2015 - 4:50am
Hi, I'm using the FFT package (the FFTW bindings), and for a complex FFT I get back : (FFTWReal r, Ix i, Shapable i) => CArray i (Complex r) -> CArray i (Complex r) at first I thought I could use elems to get to a list and then do Vector.Storable.fromList to get back to a Vector.Storable. Unfortunately r is FFTWReal so i need to map over the returned list and get a haskell Double back from that. FFTWReal maps to RealFloat which i found, but i haven't found anything that will take a RealFloat and give me a Double. Was hoping someone might know how to do that, or maybe have a little cleaner way to get from the FFT data type over to Vector. Thanks. Brian
Categories: Offsite Discussion

ANN: codec: easy bidirectional serialization andfirst-class record construction

haskell-cafe - Tue, 05/12/2015 - 3:47am
Tired of writing complementary parseJSON/toJSON, peek/poke or Binary get/put functions? codec <> provides easy bidirectional serialization of plain Haskell records in any Applicative context. All you need to do is provide a de/serializer for every record field in any order you like, and you get a de/serializer for the whole structure. The type system ensures that you provide every record exactly once. It also includes a library for general record construction in an Applicative context, of which creating codecs is just one application. *JSON:* data User = User { username :: Text , userEmail :: Text , userLanguages :: [ Text ] , userReferrer :: Maybe User } deriving Show genFields ''User userCodec :: JSONCodec User userCodec = obj "user object" $ User $>> f_username >-< "user" >>> f_userEmail >-< "email" >>> f_userLanguages >-< "languages" >>> f_userReferrer >-< opt "referrer" instance FromJSON User where parseJSON = parseVal userC
Categories: Offsite Discussion

mapM /= traverse?

libraries list - Mon, 05/11/2015 - 8:15pm
I was hoping that in GHC 7.10 we would make mapM = traverse for lists, but it appears this isn't the case: the Traversable instance for lists overrides mapM to be the manually-defined version in terms of foldr. Why is this? Fusion? Unfortunately since I want mapM = traverse (for Haxl) I'll need to continue to redefine it in our custom Prelude. Cheers, Simon
Categories: Offsite Discussion

New Functional Programming Job Opportunities

haskell-cafe - Mon, 05/11/2015 - 5:00pm
Here are some functional programming job opportunities that were posted recently: Haskell Web Engineer at Front Row Education Sr. Software Engineer at McGraw-Hill Education Cheers, Sean Murphy
Categories: Offsite Discussion

RFC: "Native -XCPP" Proposal

glasgow-user - Wed, 05/06/2015 - 12:08pm
Hello *, As you may be aware, GHC's `{-# LANGUAGE CPP #-}` language extension currently relies on the system's C-compiler bundled `cpp` program to provide a "traditional mode" c-preprocessor. This has caused several problems in the past, since parsing Haskell code with a preprocessor mode designed for use with C's tokenizer has caused already quite some problems[1] in the past. I'd like to see GHC 7.12 adopt an implemntation of `-XCPP` that does not rely on the shaky system-`cpp` foundation. To this end I've created a wiki page to describe the actual problems in more detail, and a couple of possible ways forward. Ideally, we'd simply integrate `cpphs` into GHC (i.e. "plan 2"). However, due to `cpp`s non-BSD3 license this should be discussed and debated since affects the overall-license of the GHC code-base, which may or may not be a problem to GHC's user-base (and that's what I hope this discussion will help to find out). So please go ahead and
Categories: Offsite Discussion

Looking for retainers of PINNED objects

glasgow-user - Wed, 04/29/2015 - 5:37am
Hi all, I'm profiling a fairly large program which seems to have a space leak. The heap profiling (-hc) shows that PINNED objects are accumulated over time. In order to check the retainers of the objects, I ran the retainer profiling. Unfortunately it didn't output anything with -hr -hcPINNED. Also, this is just a guess though, the retainer profiling without any filters (I mean just -hr) doesn't seem to include PINNED objects at all. Is there a way to check what retains the PINNED objects? Thanks, Mitsutoshi _______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users< at >
Categories: Offsite Discussion


glasgow-user - Sat, 04/11/2015 - 5:44pm
What’s the story with this? I tried to follow the instructions here: <> but I get Dominic Steinitz dominic< at > _______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users< at >
Categories: Offsite Discussion

New gtk2hs 0.12.4 release

gtk2hs - Wed, 11/21/2012 - 12:56pm

Thanks to John Lato and Duncan Coutts for the latest bugfix release! The latest packages should be buildable on GHC 7.6, and the cairo package should behave a bit nicer in ghci on Windows. Thanks to all!


Categories: Incoming News