News aggregator

Yesod Web Framework: Reading files from the proc filesystem

Planet Haskell - Sun, 07/27/2014 - 4:50am

I was stumped by this one myself for a bit today, so I thought writing it up in a blog post would be a good way to make sure (1) I don't forget this little fact, and (2) hopefully the next person doesn't need to puzzle over this as long as I did. Let's say you want to read the contents of a file in the proc filesystem, such as /proc/uptime. There are many ways to do that in Haskell. Let's ignore any streaming data framework for the moment, and instead focus on just the "string-like" types: String and strict/lazy ByteString/Text. Here's a little program that tries all of them out:

import qualified Data.ByteString as S import qualified Data.ByteString.Lazy as L import qualified Data.Text.IO as T import qualified Data.Text.Lazy.IO as TL test :: Show a => String -> (FilePath -> IO a) -> IO () test name reader = do contents <- reader "/proc/uptime" putStrLn $ name ++ ": " ++ show contents main :: IO () main = do test "String " readFile test "strict ByteString" S.readFile test "lazy ByteString " L.readFile test "strict Text " T.readFile test "lazy Text " TL.readFile

Given that the uptime file is just simple ASCII data, you'd probably assume (like I did) that all of these will produce the same result. In fact, that's not the case. On my system, the results are:

String : "60740.70 136144.86\n" strict ByteString: "" lazy ByteString : "60740.70 136144.86\n" strict Text : "60740.70 136144.86\n" lazy Text : "60740.70 136144.86\n"

Strict ByteString reading is returning an empty value! Why is this happening? It's actually quite easy to see once you throw in two new pieces of information. First, let's look at the implementation of Data.ByteString.readFile:

readFile :: FilePath -> IO ByteString readFile f = bracket (openBinaryFile f ReadMode) hClose (\h -> hFileSize h >>= hGet h . fromIntegral)

Notice how we allocate a buffer exactly the right size to read in the entire contents of the file. We don't do this with any of the file reading functions. For the lazy variants, we don't want to read the entire file into memory at once. And for strict Text, knowing the size of the file doesn't tell us the size of the buffer we need to allocate, due to variable length encoding. So this nifty optimization only applies to strict ByteStrings.

Now piece of data number two:

$ ls -l /proc/uptime -r--r--r-- 1 root root 0 Jul 27 13:56 /proc/uptime

Huh, the file is empty! As is well documented, virtually every file in the proc filesystem is listed as empty, and the contents are generated on demand by the kernel.

So how do you read the file contents into a strict ByteString? There are actually plenty of approaches that work. In my case, I ended up just writing a helper function using conduit:

localReadFile fp = IO.withBinaryFile fp IO.ReadMode $ \h -> sourceHandle h $$ foldC

But probably the simplest thing to do is to just convert a lazy ByteString into a strict ByteString, e.g. fmap L.toStrict . L.readFile.

Categories: Offsite Blogs

Converting Make to Shake

Haskell on Reddit - Sun, 07/27/2014 - 3:45am
Categories: Incoming News

New to Nix; problem using cabal2nix

Haskell on Reddit - Sun, 07/27/2014 - 12:53am

I've finally got round to trying out Nix.

I'm trying to set up a project sandbox, as described by /u/ocharles in How I Develop with Nix.

I'm under the impression that I should be able to generate (most of?) the default.nix file from my existing *.cabal file using cabal2nix.

However, cabal2nix isn't working for me.

$ cabal2nix ./maison.cabal % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (22) The requested URL returned error: 404 Not Found /home/dave/.nix-profile/bin/nix-prefetch-url: download of ‘http://hackage.haskell.org/packages/archive/maison/0.1.0.0/maison-0.1.0.0.tar.gz’ failed cabal2nix: Error: Cannot compute hash. (Not a hackage project?) Specify hash explicitly via --sha256 and add appropriate "src" attribute to resulting nix expression.

cabal2nix correctly concludes that my package is not on Hackage (why it had to try downloading it instead of noting the cabal file is local I don't know).

cabal2nix wants me to give it a hash. What does it want a hash of?

Have I misunderstood? Is cabal2nix only for packages released on Hackage, and not for packages under development?

submitted by dave4420
[link] [11 comments]
Categories: Incoming News

[] \\ [1..] diverges - intended?

haskell-cafe - Sat, 07/26/2014 - 8:39pm
Hi, I just noticed that import Data.List [] \\ [1..] diverges, although technically it doesn't have to (the docs suggest to me that it could just be [], and a non-diverging implementation is possible). Same for [1,2] \\ [1..] of course. Was this intended?
Categories: Offsite Discussion

Am I missing something about unfoldr?

libraries list - Sat, 07/26/2014 - 8:34pm
The current definition: unfoldr :: (b -> Maybe (a, b)) -> b -> [a] unfoldr f b = case f b of Just (a,new_b) -> a : unfoldr f new_b Nothing -> [] I haven't yet looked over the Core to see if GHC is able to do a worker/wrapper transformation to inline it, but we definitely want to inline it (in some appropriate phase), because doing so has the potential to erase the Maybes. It also isn't a good producer for foldr/build. However, if we (re)write it like this: unfoldrB f b c n = go b where go b = case f b of Just (a,new_b) -> a `c` go new_b Nothing -> n unfoldr f b = build (unfoldrB f b) then we get some really nice fusion: foldr c n (unfoldr f b) = foldr c n (build (unfoldrB f b)) = unfoldrB f b c n The list is gone, and there are no new closures or data structures in its place. As a side benefit, we could write groupBy as an unfoldr, and make it a somewhat better producer, especially when the groups are small. I don't *think* there's any way to make groupBy a good co
Categories: Offsite Discussion

Moving from understanding monad transformers to knowing when to use monad transformers?

Haskell on Reddit - Sat, 07/26/2014 - 8:06pm

As the title says, I've read the papers on monad transformers and I get them at least on a basic level, but when I'm coding I never think "this is a problem I should use a monad transformer stack for".

submitted by Hrothen
[link] [9 comments]
Categories: Incoming News

Cartesian Closed Comic: Safety

Planet Haskell - Sat, 07/26/2014 - 5:00pm

Categories: Offsite Blogs

Kototama: Playing with Haskell

Planet Haskell - Sat, 07/26/2014 - 4:00pm

After reading a few books on Haskell I had to face the reality: learning by doing a project was necessary!

I chose a project which was easy enough to be finished in a few weeks or months (but still slightly challenging) and had a practical utility.

My project is a JSON command-line utility loosely inspired by underscore-cli and Hawk. Arbitrary Lens expressions can be used to filter or transform the input.

If you don’t know what Lens are, think of them as getters/setters/filters/functions combinators similar to JQuery or CSS selectors but for any type of data-structures. I’m still a beginner regarding Lens.

The challenge for me was to learn how to dynamically evaluate Haskell expressions. This is uncommon since Haskell is statically typed. The library I used to do that is naturally limited in its functionality in comparison to a Lisp but the final result is all but disappointing.

For the purpose of my program I implemented a pretty printer for JSON similar to aeson-pretty but with colors. Maybe I should package it for Hackage?

Once I had hdevtools setup for cabal sandboxes, programming in Haskell was enjoyable. Refactoring is easy thanks to the strong type system. I was stuck once or twice with the type system but the people on the #haskell channel were helpful. The code has a certain form of esthetic even if I feel more knowledge would allow me to be cleaner. For example I wonder if it is possible to avoid pattern matching on Left and Right for multiple calls which return something like IO (Either x y), since both IO and Either are monads.

You can have a look at the project here:

https://github.com/kototama/jp

Categories: Offsite Blogs

Neil Mitchell: Converting Make to Shake

Planet Haskell - Sat, 07/26/2014 - 2:49pm

Summary: I have converted over 10,000 lines from Make to Shake. Here are some tips I learnt along the way.

Make is the de facto build system for large projects - if no one made an active choice, your project is probably using Make. The Shake build system can be a better alternative, but how should you convert? The following tips are based on my experience converting a 10,000 line Make system to Shake.

Shake can do whatever Make can

Shake is more powerful than Make, so if Make could do something, Shake probably can too. As a first approximation, the Make snippet:

output: input1 input2
shell command to run

Becomes:

"output" *> \out -> do
need ["input1","input2"]
cmd Shell "shell command to run"

In addition:

  • Variables in Make usually map to normal Haskell variables.
  • Definitions of rules and dependencies use the functions from Development.Shake. For example, .PHONY maps to the phony function.
  • Filepath manipulation uses the functions from Development.Shake.FilePath.
  • Dynamically generated include files can be handled with needMakefileDependencies from Development.Shake.Util.

Preserve the file/directory structure

The existing Make system will generate object files with particular names in particular places. Often these locations aren't what you would pick if you wrote the build system afresh. However, resist the temptation to "clean up" these pieces during the conversion. Treat the file locations as a specification, which lets you focus on the conversion to Shake without simultaneously redesigning a large and complex build system.

Treat the Makefile as a black box

Often the existing Makefile will be hard to understand, and sometimes won't be worth reading at all. The most important information in the Makefile is what commands it runs, which can be determined by make clean && make -j1 > log.txt, which captures a complete list of the commands run. From the commands it is usually relatively easy to determine the inputs and outputs, from which you can write the Shake rules. However, the Makefile can be useful to figure out which commands to group into a single rule, and how to generalise rules to cover multiple files.

Split the metadata from the logic

Often the Makefiles combine metadata (these object files go into this executable) with logic (use gcc -O2 to build all executables). Shake is great for writing build logic, but metadata is often better placed in separate files (the Haskell syntax can be a little heavy). You can use the full power of Haskell to store whatever metadata you require, and addOracle from Shake can introduce granular dependencies on the information. The module Development.Shake.Config provides some helper functions that might serve as a suitable base.

To bootstrap the Shake system, often the metadata can be extracted from the existing Makefiles. You can write a temporary script to parse the Makefile and extract whatever you consider the metadata, clean it up, and write it to new configuration files. Initially the config files are generated, but once you delete the Make original, they become source files.

Focus on a single platform/configuration

Often a build system will be cross-platform (Linux/Mac/Windows), build multiple targets (binaries/distribution package/documentation) and build multiple configurations (release/debug/profile). To start the conversion, focus only on the most heavily developed platform/configuration - if the migration is successful, abstracting over the differences is far easier in Shake than Make. You may wish to start with a simple target to try out Shake (e.g. documentation), but after that work on the target developers use every day, so that the developers can make use of the improvements sooner, motivating the migration.

Convert bottom up

Shake demands that it built all the dependencies (it checks the modification time is equal to what it remembered), in contrast Make only requires that targets are newer than their dependencies. As a result, you should start converting the leaves of the build system to Shake, and work upwards. Provided you use the same file/directory structure, you can then build what you have defined with Shake, then finish the build with Make, checking the result still works as expected.

Run Make and Shake in parallel

One you have migrated enough of the build system to be useful (the usual targets in the most common configuration), you should encourage some developers to try Shake instead of Make. These developers will find things that don't work properly, hidden features in the Make system that no one knew about etc. Expect to fix problems and iterate several times.

Hopefully the Shake system will be faster and more robust. Once these advantages have encouraged all the main developers to make the switch, you should delete/disable the Make system and expect it to bitrot quickly.

Refactor individual rules

As you are converting rules from Make to Shake you can translate them directly and refactor later, or convert straight into more idiomatic Shake. As an example, you might start with:

cmd Shell "ls >" out

The argument Shell tells Shake to use the system shell, meaning that > redirect works. Later on you may wish to switch to:

Stdout result <- cmd "ls"
writeFile' out result

Now you are invoking the ls command directly, capturing the output using Shake. Sometime later you may switch to:

getDirectoryFiles "." ["*"]

Which is the Shake tracked way of getting a list of files. Similarly, calling sed or for through Shell should probably be gradually converted to Shake/Haskell operations.

Refactor the whole

Once you have converted the whole build system, and disabled the original Make system, you may wish to refactor the build system - putting files in more appropriate places, rethinking file dependencies etc. In truth, I've never got round to this step, and I would be surprised if many people did. However, as the build system grows, hopefully the new bits with sensible decisions will gradually begin to outnumber the old bits with questionable design.

Ask if you get stuck

Build systems (even in Shake) are complex entities, with intricate coordination between files, which mostly run untyped external commands with many platform/version differences. As a result, build systems are often complex to write.

If you have a problem using Shake, just ask. If you can boil down the problem to something fairly standalone, ask on StackOverflow with the tag shake-build-system. If you are looking for more general advice, ask on the mailing list. If you succeed, write a blog post and tweet me.

Categories: Offsite Blogs

Edward Z. Yang: New theme!

Planet Haskell - Sat, 07/26/2014 - 7:58am

Hello loyal readers: Inside 206-105 has a new theme! I’m retiring Manifest, which was a pretty nice theme but (1) the text size was too small and (2) I decided I didn’t really like the fonts, I’ve reskinned my blog with a theme based on Brent Jackson’s Ashley, but ported to work on WordPress. I hope you like it, and please report any rendering snafus you might notice on older pages. Thanks!

Categories: Offsite Blogs

Help - I wrote some Haskell code, it works, but it's slower than Python. What did I do wrong?

Haskell on Reddit - Sat, 07/26/2014 - 6:50am

Hi everyone, I'm teaching myself Haskell on the occasional weekend, and recently I decided to implement an interview question. It goes like this:

A student in the University Of Ouagadougou has to reach 100 credits or more, in order to get a degree. You're given two dictionaries (hash-maps): 1) A mapping of course-id to course-credits, signifying how many credits each course provides 2) A mapping of course-id to a list of prerequisite course-ids, signifying which other courses must be taken before you could take that course. Write a program that finds the smallest set of courses that will earn you a degree.

I recognized it as a simple search problem. Because I'm a novice in Haskell, I started by writing a solution in Python, so I can wrap my head around it, and then translated it to Haskell. The resulting Haskell implementation works, produces the correct result, and is relatively terse. However, it runs slower than the Python implementation! I must be doing something wrong, but I can't figure out what it is.

Here are the two implementation:

Haskell implementation

Python implementation

(They include a similar initialization of the hash-maps, just so there's something to run on)

I would really appreciate any help, and also any other comments about my haskell code! Thanks

EDIT:

u/bgamari refactored the code into a much faster version, simply by using IntSet and HashSet. Here's his comment

submitted by erez27
[link] [43 comments]
Categories: Incoming News

ANNOUNCE: hplayground: haskell client-side webframework

haskell-cafe - Sat, 07/26/2014 - 2:34am
hplayground [1] is a haskell framework that compiles to JavaScript with the haste [5] compiler. It handles reactive effects under a applicative-monad that controls the modifications of the user interface. It is quite easy to create dynamic, composable applications using ordinary, beatiful, idiomatic haskell without special constructions. Rather than a framework, it is an EDSL. Full reinversion of control avoid spagetty event handlers and the monad confines the events and their effects to the subtree where they appear avoiding the problems of declarative reactive frameworks. There are examples running[4] the source code is in [1] And there is a todo reference application [3] running [2] too. There is also a blog post about that[7] Additionally, the syntax is almost identical to the formlet widgets in MFlow[6]. So most of the hplaygroud code could run also in a server if javascript is disabled. But MFlow and hplayground are completely independent projects. I hope that you enjoy it as much as I enjo
Categories: Offsite Discussion

Alp Mestanogullari: Write web services around databases with 0 boilerplate: announcing servant 0.1

Planet Haskell - Fri, 07/25/2014 - 6:00pm

At Zalora, we write a lot of web services and web applications in general. We use scotty a lot. And after having written a couple of web-services, despite some small handy abstractions we came up with, it really felt like we could achieve the same thing in a very concise and minimalist manner, by letting the compiler do more work for us so that we would just have to write wrapper for our SQL queries in haskell. All we had to do was to take advantage of a couple of extensions that landed in GHC in the past few years and propagate the right bits of information at the type-level. And this is what we’ve done.

The result is servant (github, hackage), which lets you declare resources, which just represent a bunch of operations (think endpoints) that operate on some type. So you could for example declare a users resource that supports adding, deleting and listing users in the following way:

mkResource "users" ctx exceptions & addWith addUser & deleteWith deleteUser & listAllWith listUsers

where:

  • ctx is just there to specify how to get our hand on our database connection for example, think of it as a withConnection function
  • exceptions is a bunch of functions that catch exceptions of various types and turns them into an error type of yours
  • addUser, deleteUser and listUsers are functions that run the corresponding SQL queries using the connection provided by ctx

And now you can turn this into a JSON-based webservice by simply applying Servant.Scotty.runResource to this simple definition. Then, provided you have written a handful of instances as required by each operation, you’ll have a small REST-y webservice with 3 endpoints that do what you expect.

The more interesting aspect of servant however is that the add, delete and listall operations just happen to be some prelude operations provided by the servant packages. You can define your own in just the same way the standard ones are defined. The same applies to the automatic JSON-based request body/response body handling or to the web-framework backend used (we only have a scotty one for now but you could write your own for any other framework by drawing some inspiration from the scotty one). You can extend servant in basically every possible direction.

If you want to learn more about servant, how it can be used and how it works, you may be interested to check out the README from github which contains some documentation links, that I’ll reproduce here:

  • Getting started with servant, which guides you through building the simple webservice we’ve seen above. There’s an example in the repository with the code covered in this getting started guide, with a cabal file and everything.
  • Tutorial, which dives much more into servant’s packages and modules and its inner workings, with some illustrations of the extensibility of servant.
  • Haddocks for all servant packages

We would of course be glad to hear any kind of feedback, so please do not hesitate to shoot us an email with comments, and report any issue you may encounter on our github.

Posted on July 26, 2014
Categories: Offsite Blogs

Proof that scanl' is fusion-safe

libraries list - Fri, 07/25/2014 - 5:07pm
This proof is horribly inefficient. I imagine it could be fixed by appealing to the universal property of foldr or something like that, but I don't have time to look at that right now, so I'll just show you what I have right now; I'm hoping someone will be willing to verify that I haven't made any major errors. scanl' :: (b -> a -> b) -> b -> [a] -> [b] scanl' f a bs = build $ \c n -> a `seq` a `c` foldr (\b g x -> let b' = f x b in b' `seq` (b' `c` g b')) (\b -> b `seq` n) bs a
Categories: Offsite Discussion