News aggregator

Threads and critical sections (threadWaitRead)

haskell-cafe - Fri, 05/22/2015 - 12:55pm
Hello cafe, I'm currently trying to understand Haskell's threading model and GHC's implementation of it. Specifically I wondered about the safety of file IO. Consider this piece of code: Context: - The socket is configured non-blocking. - No other operation will hold the MVar over a blocking call (because this would make it impossible to close it while it's blocking). - My close operation will fill the MVar with -1 after it's closed and will hold the MVar during the close operation. Questions: - Is it theoretically possible that my thread gets paused at (1) or within threadWaitRead, another thread closes the socket (or even worse opens another file with the same fd) _and_ my thread is resumed afterwards (I'm not asking about async exceptions)? - What constructs contain safepoints and which are guaranteed to be executed without interruption? Considerations (so far): - It is unlikely that (1) is a safepoint as no memory allocations occur, but is it guaranteed? - If the socket were closed dur
Categories: Offsite Discussion

error in 7.10.1 ?

glasgow-user - Fri, 05/22/2015 - 12:26pm
Dear GHC developers, The confirmation token letter from the bug tracker travels too long, I do not know, why. Meanwhile I cannot report there. So, I report it in the below way: http://www.botik.ru/pub/local/Mechveliani/ghcQuest/7.10.1-errReport-may22-2015.zip Make it, please, with ghc-7.10.1 as it is written there in install.txt, including making demotest/Main. Then ./Main breaks in the middle reporting a DoCon application error ---------------------------------------------------------------------------- Factorization in K1[x], K1 a generic finite extension of a prime field K = Z/(p) .. factor f, f = (x^10 + x^7 + x^5 + x^3 + x^2 + (2)*x+ (2)) <- K[x] = ((Z/<5>)["t"]/<(t^3 + 4*t + 3)>)["x"] Domain term of K must contain a Finite field description. --------------------------------------------------------------------------- I obtain this for ghc-7.10.1 made from source by ghc-7.8.2 on Debian Linux, x-86, 64 bit. The story is as follows. docon-2.12 http://www.bot
Categories: Offsite Discussion

-Wall and the fail method

glasgow-user - Fri, 05/22/2015 - 11:06am
Can -Wall be extended to report pattern match failures in do expressions, like it does for case expressions? Prelude> :set -Wall Prelude> let f = do Just x <- return Nothing; return x Prelude> let g = case Nothing of Just x -> x <interactive>:9:9: Warning: Pattern match(es) are non-exhaustive In a case alternative: Patterns not matched: Nothing One can argue that it's similar to undefined, error, and various unsafeSomething functions, which I think should be reported as well, if possible. But these things can be found already with a simple grep while a pattern match cannot. I bet it has been discussed already, but "fail" is a terrible search term, so I cannot find anything relevant in the archives nor in the bug tracker.
Categories: Offsite Discussion

open gl type conversions

haskell-cafe - Fri, 05/22/2015 - 6:13am
Following code GLFW.setWindowSizeCallback w (Just (\window w h -> do GL.viewport $= (GL.Position 0 0, GL.Size w h) emo1.hs:42:50: Couldn't match type ‘Int’ with ‘Foreign.C.Types.CInt’ Expected type: GLsizei Actual type: Int In the first argument of ‘Size’, namely ‘w’ In the expression: Size w h I can never seem to figure out if i really need to do a type conversion, or just the make the appropriate type declarations. Sometimes it seems to be a combination of both. Part of the complication here is that w and h are, in fact, `Int` because that's there type in the type signature of WindowSizeCallback (Window -> Int -> Int -> IO () ). GL.Size really needs a CInt so it seems like I have to do some sort of explicit conversion, but I can never seem to figure out exactly how to find the necessary function. Humorously i tried putting Int->CInt in hoogle, and the only thing that looked like it might be useful was unsafeCoerce, but I'm pretty sure that's not t
Categories: Offsite Discussion

Library design around FFI library

Haskell on Reddit - Fri, 05/22/2015 - 5:12am

I'm implementing the low-level interface to a large C library using c2hs. First I'll briefly describe the low-level setting and then I'll ask a few questions concerning the design of the higher-level interface (I've asked the same q. on haskell-cafe so feel free to reply here or there).

The C functions can be broadly classified with a few criteria:

  • the main datastructure they operate on, e.g. returnType AFunction (A* a, ...);

  • creation, modification, read access, destruction operations for each of these datastructures, following a common pattern, e.g.: ACreate(), ASet1(), ASet2(), .., ARead(), ADestroy(), BCreate(), etc. The library requires explicit destruction of all Create()d objects.

The library rests on top of MPI, so there are e.g. AScatterBegin(), AScatterEnd() function pairs that account for potentially large delays associated with computing and moving data around.

On the Haskell side, the C pointer types are represented (using GeneralizedNewtypeDeriving and StandaloneDeriving) as

newtype Ah = Ah (Ptr Ah) deriving instance Storable Ah

and e.g. a Create() function returns the representation of the fresh pointer along with the usual integer error code:

ahCreate :: MPIComm -> IO (Ah, Err) Questions:

What abstraction may I use to capture logic such as:

  • separate read-only functions from the potentially overwriting ones

  • guarantee precedence relations (e.g. computation can only occur after initialization)

  • collect or even compose error codes, letting computation go through only if Err==0 (e.g. a n-ary Maybe ?) Sequencing in the IO monad causes all effectful functions to run and return their result and/or error code.

  • reflect the deallocation of a C-land object on the Hs side.

All feedback is welcome; Thank you in advance

submitted by ocramz
[link] [7 comments]
Categories: Incoming News

What will happen to Haskell in the future?

Haskell on Reddit - Fri, 05/22/2015 - 3:31am

I mean, Haskell is not very popular yet, even though it is an awesome and fast language. Do you think Haskell can be a big deal in the future or not? Does OOP offer more possibilities? Or will they both coexist just as they do now? I just wonder if there clearly is a "winner" between OOP and FP.

submitted by sammecs
[link] [39 comments]
Categories: Incoming News

The new Stackage Server

Haskell on Reddit - Fri, 05/22/2015 - 2:17am
Categories: Incoming News

FP Complete: The new Stackage Server

Planet Haskell - Fri, 05/22/2015 - 1:30am

tl;dr Please check out beta.stackage.org

I made the first commit to the Stackage Server code base a little over a year ago. The goal was to provide a place to host package sets which both limited the number of packages from Hackage available, and modified packages where necessary. This server was to be populated by regular Stackage builds, targeted at multiple GHC versions, and consisted of both inclusive and exclusive sets. It also allowed interested individuals to create their own package sets.

If any of those details seem surprising today, they should. A lot has happened for the Stackage project in the past year, making details of what was initially planned irrelevant, and making other things (like hosting of package documentation) vital. We now have LTS Haskell. Instead of running with multiple GHC versions, we have Stackage Nightly which is targeted at a single GHC major version. To accomodate goals for GPS Haskell (which unfortunately never materialized), Stackage no longer makes corrections to upstream packages.

I could go into lots more detail on what is different in project requirements. Instead, I'll just summarize: I've been working on a simplified version of the Stackage Server codebase to address our goals better, more easily ensure high availability, and make the codebase easier to maintain. We also used this opportunity to test out a new hosting system our DevOps team put together. The result is running on beta.stackage.org, and will replace the official stackage.org after a bit more testing (which I hope readers will help with).

The code

All of this code lives on the simpler branch of the stackage-server code base, and much to my joy, resulted in quite a bit less code. In fact, there's just about a 2000 line reduction. The rest of this post will get into how that happened.

No more custom package sets

One of the features I mentioned above was custom package sets. This fell out automatically from the initial way Stackage Server was written, so it was natural to let others create package sets of their own. However, since release, only one person actually used that feature. I discussed with him, and he agreed with the decision to deprecate and then remove that functionality.

So why get rid of it now? Two powerful reasons:

  • We already host a public mirror of all packages on S3. Since we no longer patch upstream packages, it's best if tooling is able to just refer to that high-reliability service.
  • We now have Git repositories for all of LTS Haskell and Stackage Nightly. Making these the sources of package sets means we don't have two (possibly conflicting) sources of data. That brings me to the second point
Upload code is gone

We had some complicated logic to allow users to upload package sets. It started off simple, but over time we added Haddock hosting and other metadata features, making the code more complex. Actually, it ended up having two parallel code paths for this. So instead, we now just upload information on the package sets to the Git repositories, and leave it up to a separate process (described below) to clone these repositories and make the data available to the server.

Haddocks on S3

After generating a snapshot, the Haddocks used to be tarred and compressed, and then uploaded as a compressed bundle to S3. Then, Stackage Server would receive a request for files, unpack them, and serve them. This presented some problems:

  • Users would have to wait for a first request to succeed during the unpacking
  • With enough snapshots being generated, we would eventually run out of disk space and need to clear our temp directory
  • Since we run our cluster in a high availabilty mode with multiple horizontally-scaled machines, one machine may have finished unpacking when another didn't, resulting in unstyled content (see issue #82).

Instead, we now just upload the files to S3 and redirect there from stackage-server (though we'll likely switch to reverse proxying to allow for nicer SSL urls). In fact, you can easily view these docs, at URLs such as http://haddock.stackage.org/lts-2.9/ or https://s3.amazonaws.com/haddock.stackage.org/nightly-2015-05-21/index.html.

These Haddocks are publicly available, and linkable from projects beyond Stackage Server. Each set of Haddocks is guaranteed to have consistent internal links to other compatible packages. And while some documentation doesn't generate due to known package bugs, the generation is otherwise reliable.

I've already offered access to these docs to Duncan for usage on Hackage, and hope that will improve the experience for users there.

Metadata SQLite database

Previously, information on snapshots was stored in a PostgreSQL database that was maintained by Stackage Server. This database also had package metadata, like author, homepage, and description. Now, we have a completely different process:

  • The all-cabal-metadata from the Commercial Haskell Special Interest Group provides an easily cloneable Git repo with package metadata, which is automatically updated by Travis.
  • We run a cron job on the stackage-build server that updates the lts-haskell, stackage-nightly, and all-cabal-metadata repos and generates a SQLite database from them with all of the data that Stackage Server needs. You can look at the Stackage.Database module for some ideas of what this consists of. That database gets uploaded to Amazon S3, and is actually publicly available if you want to poke at it
  • The live server downloads a new version of this file on a regular basis

I've considered spinning off the Stackage.Download code into its own repository so that others can take advantage of this functionality in different contexts if desired. Let me know if you're interested.

At this point, the PostgreSQL database is just used for non-critical functionality, such as social features (tags and likes).

Slightly nicer URLs

When referring to a snapshot, there are "official" short names (slugs), of the form lts-2.9 and nightly-2015-05-22. The URLs on the new server now reflect this perfectly, e.g.: https://beta.stackage.org/nightly-2015-05-22. We originally used hashes of the snapshot content for the original URLs, but that was fixed a while ago. Now that we only have to support these official snapshots, we can always (and exclusively) use these short names.

As a convenience, if you visit the following URLs, you get automatic redirects:

  • /nightly redirects to the most recent nightly
  • /lts to the latest LTS
  • /lts-X to the latest LTS in the X.* major version (e.g., today, /lts-2 redirects to /lts-2.9)

This also works for URLs under that hierarchy. For example, consider https://beta.stackage.org/lts/cabal.config, which is an easy way to get set up with LTS in your project (by running wget https://beta.stackage.org/lts/cabal.config).

ECS-based hosting

While not a new feature of the server itself, the hosting cluster we're running this on is brand new. Amazon recently released EC2 Container Service, which is a service for running Docker containers. Since we're going to be using this for the new School of Haskell, it's nice to be giving it a serious usage now. We also make extensive use of Docker for customer projects, both for builds and hosting, so it's a natural extension for us.

This ECS cluster uses standard Amazon services like Elastic Load Balancer (ELB) and auto-scaling to provide for high availability in the case of machine failure. And while we have a lot of confidence in our ability to keep Stackage Server up and running regularly, it's nice that our most important user-facing content is provided by these external services:

This provides for a pleasant experience in both browsing the website and using Stackage in your build system.

A special thanks to Jason Boyer for providing this new hosting cluster, which the whole FP Complete team is looking forward to putting through its paces.

Categories: Offsite Blogs

Advice on State, Map, and Hashes

Haskell on Reddit - Thu, 05/21/2015 - 11:45pm

Hi guys, I'm learning Haskell by doing Project Euler problems. I've gotten the hang of using the State monad along with Map when I would normally use a hash table. But now I need two separate hash tables, and as far as I can tell State only allows one persistent data structure. For now, I can make a Map that stores an extra Bool to differentiate, but what is the real way to do this? Thanks in advance for any advice.

submitted by peterlew
[link] [9 comments]
Categories: Incoming News

Increasing parallelism in a frequency counter

haskell-cafe - Thu, 05/21/2015 - 10:08pm
Hi everyone, I've started playing with some parallel programming in Haskell, mostly by applying the first few examples from Simon Marlow's book to my own toy problem: https://github.com/apauley/parallel-frequency I'm quite pleased with the speedup I have so far, the fastest parallel implementation is just over twice as fast as its sequential counterpart. But looking at the threadscope images from the generated event logs I can see that there are still significant portions of my program that runs on only one core. Is anyone interested to see if we can increase the parallelism in this frequency counter? Kind regards, Andreas -- https://keybase.io/apauley http://za.linkedin.com/in/apauley http://www.meetup.com/lambda-luminaries http://www.meetup.com/Bitcoins-Baby-Bitcoins GPG Public Key: https://keybase.io/apauley/key.asc
Categories: Offsite Discussion

MonadFix instance for ParsecT

Haskell on Reddit - Thu, 05/21/2015 - 9:47pm

Howdy,

I've been scratching my head for a while coming up with a proper instance of MonadFix for ParsecT (or even Parsec, assuming -XTypeSynonymInstances.) That is:

instance Monad m => MonadFix (ParsecT s u m) where mfix f = ???

Any help would be appreciated. Thank you!

submitted by analphabetic
[link] [2 comments]
Categories: Incoming News

Does accelerate-llvm make repa obsolete?

Haskell on Reddit - Thu, 05/21/2015 - 9:28pm

Now that accelerate can run computations on the cpu (link) and performs better, is there still a need for repa?

submitted by precalc
[link] [4 comments]
Categories: Incoming News

documentation links broken in 7.10.1

glasgow-user - Thu, 05/21/2015 - 6:49pm
It seems like haddock's index.html generation is broken in 7.10.1. Namely, it creates links to e.g. Codec-Binary-UTF8-Generic.html when it should be to utf8-string-1/html/Codec-Binary-UTF8-Generic.html This is haddock 2.16.0, which comes with the binary distribution for 7.10.1. I looked at https://github.com/haskell/haddock/issues but didn't see anything related to this, which makes me think it's just me. You'd think someone would notice if all documentation links were broken. Does anyone else see this?
Categories: Offsite Discussion

Why is cabal such sewage?

Haskell on Reddit - Thu, 05/21/2015 - 5:48pm

Why on earth is cabal so miserable to work with? Why can't I uninstall packages without having to nuke my installation? No, I don't care if you say it's not a package manager, its installing shit on my system, therefore its a package manager. I love coding haskell, but cabal is so shit and worthless. I'd rather code in Java than dick around in Haskell, and I hate Java. If anything is going to kill Haskell, its this trash.

submitted by dpakattack
[link] [11 comments]
Categories: Incoming News

23% of calls to map fuse away

haskell-cafe - Thu, 05/21/2015 - 3:56pm
Hi, I just compiled all¹ of Stackage LTS 2.9 with GHC 7.8.4 with -ddump-rule-firings, and counted the occurrences of the rules. Here is a fun fact: 23% of all (static) occurrences of "map" fuse away². And 50387 lists are fused away in total³ Full data at https://gist.github.com/nomeata/071c1f87450cf668bbeb Greetings, Joachim ¹ but the SDL2 bindings ² 16925 × rule map 12904 × rule mapList, which fires if no fusion happened. ³ 27846 × rule fold/build 11293 × rule foldr/single 11248 × rule augment/build
Categories: Offsite Discussion

overlapping instances 7.10.1

glasgow-user - Thu, 05/21/2015 - 2:40pm
People, I wrote recently about finding places to set {-# OVERLAPPING #-} when porting an application from 7.8.2 to 7.10.1. I am doing this for porting docon-2.12 from 7.8.2 to 7.10.1. And ghc-7.10.1 indeed helped me to find several places to set this pragma (instead of using the total key -XOverlappingInstances). Finally, it has come to this module: --------------------------------------------- Preprocessing library docon-2.12.1... [48 of 86] Compiling ResRing__ ( ResRing__.hs, dist/build/ResRing__.o ) ResRing__.hs:183:31: Overlapping instances for Eq (Maybe PropValue) arising from a use of ‘/=’ Matching instances: instance Eq a => Eq (Maybe a) -- Defined in ‘GHC.Base’ instance (Residue r, Eq a) => Eq (r a) -- Defined in ‘ResEuc0_’ In the expression: lookup IsGxBasis ps /= Just Yes ... ---------------------------------------------- As before, I set instance {-# OVERLAPPING #-} (Residue r, Eq a) => Eq (r a) where ... in ResEuc0_.hs.
Categories: Offsite Discussion

Neil Mitchell: Handling Control-C in Haskell

Planet Haskell - Thu, 05/21/2015 - 2:19pm

Summary: The development version of ghcid seemed to have some problems with terminating when Control-C was hit, so I investigated and learnt some things.

Given a long-running/interactive console program (e.g. ghcid), when the user hits Control-C/Ctrl-C the program should abort. In this post I'll describe how that works in Haskell, how it can fail, and what asynchronous exceptions have to do with it.

What happens when the user hits Ctrl-C?

When the user hits Ctrl-C, GHC raises an async exception of type UserInterrupt on the main thread. This happens because GHC installs an interrupt handler which raises that exception, sending it to the main thread with throwTo. If you install your own interrupt handler you won't see this behaviour and will have to handle Ctrl-C yourself.

There are reports that if the user hits Ctrl-C twice the runtime will abort the program. In my tests, that seems to be a feature of the shell rather than GHC itself - in the Windows Command Prompt no amount of Ctrl-C stops an errant program, in Cygwin a single Ctrl-C works.

What happens when the main thread receives UserInterrupt?

There are a few options:

  • If you are not masked and there is no exception handler, the thread will abort, which causes the whole program to finish. This behaviour is the desirable outcome if the user hits Ctrl-C.
  • If you are running inside an exception handler (e.g. catch or try) which is capable of catching UserInterrupt then the UserInterrupt exception will be returned. The program can then take whatever action it wishes, including rethrowing UserInterrupt or exiting the program.
  • If you are running with exceptions masked, then the exception will be delayed until you stop being masked. The most common way of running while masked is if the code is the second argument to finally or one of the first two arguments to bracket. Since Ctrl-C will be delayed while the program is masked, you should only do quick things while masked.
How might I lose UserInterrupt?

The easiest way to "lose" a UserInterrupt is to catch it and not rethrow it. Taking a real example from ghcid, I sometimes want to check if two paths refer to the same file, and to make that check more robust I call canonicalizePath first. This function raises errors in some circumstances (e.g. the directory containing the file does not exist), but is inconsistent about error conditions between OS's, and doesn't document its exceptions, so the safest thing is to write:

canonicalizePathSafe :: FilePath -> IO FilePath
canonicalizePathSafe x = canonicalizePath x `catch`
\(_ :: SomeException) -> return x

If there is any exception, just return the original path. Unfortunately, the catch will also catch and discard UserInterrupt. If the user hits Ctrl-C while canonicalizePath is running the program won't abort. The problem is that UserInterrupt is not thrown in response to the code inside the catch, so ignoring UserInterrupt is the wrong thing to do.

What is an async exception?

In Haskell there are two distinct ways to throw exceptions, synchronously and asynchronously.

  • Synchronous exceptions are raised on the calling thread, using functions such as throw and error. The point at which a synchronous exception is raised is explicit and can be relied upon.
  • Asynchronous exceptions are raised by a different thread, using throwTo and a different thread id. The exact point at which the exception occurs can vary.
How is the type AsyncException related?

In Haskell, there is a type called AsyncException, containing four exceptions - each special in their own way:

  • StackOverflow - the current thread has exceeded its stack limit.
  • HeapOverflow - never actually raised.
  • ThreadKilled - raised by calling killThread on this thread. Used when a programmer wants to kill a thread.
  • UserInterrupt - the one we've been talking about so far, raised on the main thread by the user hitting Ctrl-C.

While these have a type AsyncException, that's only a hint as to their intended purpose. You can throw any exception either synchronously or asynchronously. In our particular case of caonicalizePathSafe, if canonicalizePath causes a StackOverflow, we probably are happy to take the fallback case, but likely the stack was already close to the limit and will occur again soon. If the programmer calls killThread that thread should terminate, but in ghcid we know this thread won't be killed.

How can I catch avoid catching async exceptions?

There are several ways to avoid catching async exceptions. Firstly, since we expect canonicalizePath to complete quickly, we can just mask all async exceptions:

canonicalizePathSafe x = mask_ $
canonicalizePath x `catch` \(_ :: SomeException) -> return x

We are now guaranteed that catch will not receive an async exception. Unfortunately, if canonicalizePath takes a long time, we might delay Ctrl-C unnecessarily.

Alternatively, we can catch only non-async exceptions:

canonicalizePathSafe x = catchJust
(\e -> if async e then Nothing else Just e)
(canonicalizePath x)
(\_ -> return x)

async e = isJust (fromException e :: Maybe AsyncException)

We use catchJust to only catch exceptions which aren't of type AsyncException, so UserInterrupt will not be caught. Of course, this actually avoids catching exceptions of type AsyncException, which is only related to async exceptions by a partial convention not enforced by the type system.

Finally, we can catch only the relevant exceptions:

canonicalizePathSafe x = canonicalizePath x `catch`
\(_ :: IOException) -> return x

Unfortunately, I don't know what the relevant exceptions are - on Windows canonicalizePath never seems to throw an exception. However, IOException seems like a reasonable guess.

How to robustly deal with UserInterrupt?

I've showed how to make canonicalizePathSafe not interfere with UserInterrupt, but now I need to audit every piece of code (including library functions I use) that runs on the main thread to ensure it doesn't catch UserInterrupt. That is fragile. A simpler alternative is to push all computation off the main thread:

import Control.Concurrent.Extra
import Control.Exception.Extra

ctrlC :: IO () -> IO ()
ctrlC act = do
bar <- newBarrier
forkFinally act $ signalBarrier bar
either throwIO return =<< waitBarrier bar

main :: IO ()
main = ctrlC $ ... as before ...

We are using the Barrier type from my previous blog post, which is available from the extra package. We create a Barrier, run the main action on a forked thread, then marshal completion/exceptions back to the main thread. Since the main thread has no catch operations and only a few (audited) functions on it, we can be sure that Ctrl-C will quickly abort the program.

Using version 1.1.1 of the extra package we can simplify the code to ctrlC = join . onceFork.

What about cleanup?

Now we've pushed most actions off the main thread, any finally sections are on other threads, and will be skipped if the user hits Ctrl-C. Typically this isn't a problem, as program shutdown automatically cleans all non-persistent resources. As an example, ghcid spawns a copy of ghci, but on shutdown the pipes are closed and the ghci process exits on its own. If we do want robust cleanup of resources such as temporary files we would need to run the cleanup from the main thread, likely using finally.

Should async exceptions be treated differently?

At the moment, Haskell defines many exceptions, any of which can be thrown either synchronously or asynchronously, but then hints that some are probably async exceptions. That's not a very Haskell-like thing to do. Perhaps there should be a catch which ignores exceptions thrown asynchronously? Perhaps the sync and async exceptions should be of different types? It seems unfortunate that functions have to care about async exceptions as much as they do.

Combining mask and StackOverflow

As a curiosity, I tried to combine a function that stack overflows (using -O0) and mask. Specifically:

main = mask_ $ print $ foldl (+) 0 [1..1000000]

I then ran that with +RTS -K1k. That prints out the value computed by the foldl three times (seemingly just a buffering issue), then fails with a StackOverflow exception. If I remove the mask, it just fails with StackOverflow. It seems that by disabling StackOverflow I'm allowed to increase my stack size arbitrarily. Changing print to appendFile causes the file to be created but not written to, so it seems there are oddities about combining these features.

Disclaimer

I'm certainly not an expert on async exceptions, so corrections welcome. All the above assumes compiling with -threaded, but most applies without -threaded.

Categories: Offsite Blogs