After reading a few books on Haskell I had to face the reality: learning by doing a project was necessary!
I chose a project which was easy enough to be finished in a few weeks or months (but still slightly challenging) and had a practical utility.
If you don’t know what Lens are, think of them as getters/setters/filters/functions combinators similar to JQuery or CSS selectors but for any type of data-structures. I’m still a beginner regarding Lens.
The challenge for me was to learn how to dynamically evaluate Haskell expressions. This is uncommon since Haskell is statically typed. The library I used to do that is naturally limited in its functionality in comparison to a Lisp but the final result is all but disappointing.
For the purpose of my program I implemented a pretty printer for JSON similar to aeson-pretty but with colors. Maybe I should package it for Hackage?
Once I had hdevtools setup for cabal sandboxes, programming in Haskell was enjoyable. Refactoring is easy thanks to the strong type system. I was stuck once or twice with the type system but the people on the #haskell channel were helpful. The code has a certain form of esthetic even if I feel more knowledge would allow me to be cleaner. For example I wonder if it is possible to avoid pattern matching on Left and Right for multiple calls which return something like IO (Either x y), since both IO and Either are monads.
You can have a look at the project here:
Summary: I have converted over 10,000 lines from Make to Shake. Here are some tips I learnt along the way.
Make is the de facto build system for large projects - if no one made an active choice, your project is probably using Make. The Shake build system can be a better alternative, but how should you convert? The following tips are based on my experience converting a 10,000 line Make system to Shake.
Shake can do whatever Make can
Shake is more powerful than Make, so if Make could do something, Shake probably can too. As a first approximation, the Make snippet:output: input1 input2
shell command to run
Becomes:"output" *> \out -> do
cmd Shell "shell command to run"
- Variables in Make usually map to normal Haskell variables.
- Definitions of rules and dependencies use the functions from Development.Shake. For example, .PHONY maps to the phony function.
- Filepath manipulation uses the functions from Development.Shake.FilePath.
- Dynamically generated include files can be handled with needMakefileDependencies from Development.Shake.Util.
Preserve the file/directory structure
The existing Make system will generate object files with particular names in particular places. Often these locations aren't what you would pick if you wrote the build system afresh. However, resist the temptation to "clean up" these pieces during the conversion. Treat the file locations as a specification, which lets you focus on the conversion to Shake without simultaneously redesigning a large and complex build system.
Treat the Makefile as a black box
Often the existing Makefile will be hard to understand, and sometimes won't be worth reading at all. The most important information in the Makefile is what commands it runs, which can be determined by make clean && make -j1 > log.txt, which captures a complete list of the commands run. From the commands it is usually relatively easy to determine the inputs and outputs, from which you can write the Shake rules. However, the Makefile can be useful to figure out which commands to group into a single rule, and how to generalise rules to cover multiple files.
Split the metadata from the logic
Often the Makefiles combine metadata (these object files go into this executable) with logic (use gcc -O2 to build all executables). Shake is great for writing build logic, but metadata is often better placed in separate files (the Haskell syntax can be a little heavy). You can use the full power of Haskell to store whatever metadata you require, and addOracle from Shake can introduce granular dependencies on the information. The module Development.Shake.Config provides some helper functions that might serve as a suitable base.
To bootstrap the Shake system, often the metadata can be extracted from the existing Makefiles. You can write a temporary script to parse the Makefile and extract whatever you consider the metadata, clean it up, and write it to new configuration files. Initially the config files are generated, but once you delete the Make original, they become source files.
Focus on a single platform/configuration
Often a build system will be cross-platform (Linux/Mac/Windows), build multiple targets (binaries/distribution package/documentation) and build multiple configurations (release/debug/profile). To start the conversion, focus only on the most heavily developed platform/configuration - if the migration is successful, abstracting over the differences is far easier in Shake than Make. You may wish to start with a simple target to try out Shake (e.g. documentation), but after that work on the target developers use every day, so that the developers can make use of the improvements sooner, motivating the migration.
Convert bottom up
Shake demands that it built all the dependencies (it checks the modification time is equal to what it remembered), in contrast Make only requires that targets are newer than their dependencies. As a result, you should start converting the leaves of the build system to Shake, and work upwards. Provided you use the same file/directory structure, you can then build what you have defined with Shake, then finish the build with Make, checking the result still works as expected.
Run Make and Shake in parallel
One you have migrated enough of the build system to be useful (the usual targets in the most common configuration), you should encourage some developers to try Shake instead of Make. These developers will find things that don't work properly, hidden features in the Make system that no one knew about etc. Expect to fix problems and iterate several times.
Hopefully the Shake system will be faster and more robust. Once these advantages have encouraged all the main developers to make the switch, you should delete/disable the Make system and expect it to bitrot quickly.
Refactor individual rules
As you are converting rules from Make to Shake you can translate them directly and refactor later, or convert straight into more idiomatic Shake. As an example, you might start with:cmd Shell "ls >" out
The argument Shell tells Shake to use the system shell, meaning that > redirect works. Later on you may wish to switch to:Stdout result <- cmd "ls"
writeFile' out result
Now you are invoking the ls command directly, capturing the output using Shake. Sometime later you may switch to:getDirectoryFiles "." ["*"]
Which is the Shake tracked way of getting a list of files. Similarly, calling sed or for through Shell should probably be gradually converted to Shake/Haskell operations.
Refactor the whole
Once you have converted the whole build system, and disabled the original Make system, you may wish to refactor the build system - putting files in more appropriate places, rethinking file dependencies etc. In truth, I've never got round to this step, and I would be surprised if many people did. However, as the build system grows, hopefully the new bits with sensible decisions will gradually begin to outnumber the old bits with questionable design.
Ask if you get stuck
Build systems (even in Shake) are complex entities, with intricate coordination between files, which mostly run untyped external commands with many platform/version differences. As a result, build systems are often complex to write.
If you have a problem using Shake, just ask. If you can boil down the problem to something fairly standalone, ask on StackOverflow with the tag shake-build-system. If you are looking for more general advice, ask on the mailing list. If you succeed, write a blog post and tweet me.
Hello loyal readers: Inside 206-105 has a new theme! I’m retiring Manifest, which was a pretty nice theme but (1) the text size was too small and (2) I decided I didn’t really like the fonts, I’ve reskinned my blog with a theme based on Brent Jackson’s Ashley, but ported to work on WordPress. I hope you like it, and please report any rendering snafus you might notice on older pages. Thanks!
Hi everyone, I'm teaching myself Haskell on the occasional weekend, and recently I decided to implement an interview question. It goes like this:A student in the University Of Ouagadougou has to reach 100 credits or more, in order to get a degree. You're given two dictionaries (hash-maps): 1) A mapping of course-id to course-credits, signifying how many credits each course provides 2) A mapping of course-id to a list of prerequisite course-ids, signifying which other courses must be taken before you could take that course. Write a program that finds the smallest set of courses that will earn you a degree.
I recognized it as a simple search problem. Because I'm a novice in Haskell, I started by writing a solution in Python, so I can wrap my head around it, and then translated it to Haskell. The resulting Haskell implementation works, produces the correct result, and is relatively terse. However, it runs slower than the Python implementation! I must be doing something wrong, but I can't figure out what it is.
Here are the two implementation:
(They include a similar initialization of the hash-maps, just so there's something to run on)
I would really appreciate any help, and also any other comments about my haskell code! Thanks
u/bgamari refactored the code into a much faster version, simply by using IntSet and HashSet. Here's his commentsubmitted by erez27
[link] [43 comments]
At Zalora, we write a lot of web services and web applications in general. We use scotty a lot. And after having written a couple of web-services, despite some small handy abstractions we came up with, it really felt like we could achieve the same thing in a very concise and minimalist manner, by letting the compiler do more work for us so that we would just have to write wrapper for our SQL queries in haskell. All we had to do was to take advantage of a couple of extensions that landed in GHC in the past few years and propagate the right bits of information at the type-level. And this is what we’ve done.
The result is servant (github, hackage), which lets you declare resources, which just represent a bunch of operations (think endpoints) that operate on some type. So you could for example declare a users resource that supports adding, deleting and listing users in the following way:mkResource "users" ctx exceptions & addWith addUser & deleteWith deleteUser & listAllWith listUsers
- ctx is just there to specify how to get our hand on our database connection for example, think of it as a withConnection function
- exceptions is a bunch of functions that catch exceptions of various types and turns them into an error type of yours
- addUser, deleteUser and listUsers are functions that run the corresponding SQL queries using the connection provided by ctx
And now you can turn this into a JSON-based webservice by simply applying Servant.Scotty.runResource to this simple definition. Then, provided you have written a handful of instances as required by each operation, you’ll have a small REST-y webservice with 3 endpoints that do what you expect.
The more interesting aspect of servant however is that the add, delete and listall operations just happen to be some prelude operations provided by the servant packages. You can define your own in just the same way the standard ones are defined. The same applies to the automatic JSON-based request body/response body handling or to the web-framework backend used (we only have a scotty one for now but you could write your own for any other framework by drawing some inspiration from the scotty one). You can extend servant in basically every possible direction.
If you want to learn more about servant, how it can be used and how it works, you may be interested to check out the README from github which contains some documentation links, that I’ll reproduce here:
- Getting started with servant, which guides you through building the simple webservice we’ve seen above. There’s an example in the repository with the code covered in this getting started guide, with a cabal file and everything.
- Tutorial, which dives much more into servant’s packages and modules and its inner workings, with some illustrations of the extensibility of servant.
- Haddocks for all servant packages
We would of course be glad to hear any kind of feedback, so please do not hesitate to shoot us an email with comments, and report any issue you may encounter on our github.Posted on July 26, 2014
I'm looking for an API for lazy streaming of monadic data structures. I.e., I need to convert a datastructure into something that can traverse it partially and accumulate transformations instead of applying them. To be more specific I want to extend the API of the "stm-containers" library, with a function which will stream the elements of a container in such a way.
Now, of course, it could be implemented using a library like "pipes", e.g. with a function having a signature like the following:producer :: Map k v -> Producer (k, v) STM ()
But it obviously would be an overkill, which would bind the library to a specific IO streaming ecosystem.
What I'm looking for is a simpler streaming API, using which I could implement a function like the following:stream :: Map k v -> Stream STM (k, v)
And then to be able to lazily manipulate streams (filter, map, join - what not). I guess, this is what the ListT monad transformer is supposed to do, but I never dived into it and I know that it's notorious for being broken. The "pipes" library claims to provide a proper version of this transformer, but then again it binds the user to the "pipes" ecosystem.
So my question is: is there a library that solves this problem or am I missing something evident in the approach to my problem?submitted by nikita-volkov
[link] [30 comments]
Lets say you have the type class:class Foo a where f :: a -> String g :: a -> Int k :: a -> Bool
So none of the functions are defined by default. Yet I can do the following:module MyMod (Foo(f)) where ...
And it compiles! Not even a warning? People can import this module, make an instance for their types yet only be able to define f. Then when they compile their code, they'll get the warning that k and g are not defined. Yet they can't possibly define k or g!
How come ghc/haskell allows this? Is there any instance where this would be desirable? I would think that it shouldn't allow you to partially export functions in a typeclass if those functions don't have a default implementation.submitted by Die-Nacht
[link] [10 comments]
I've been trying to push management in the Haskell direction for development and while I could address most concerns there were a few that seem like deal breakers (here are the two main ones):
Memory allocation - real time safety critical code usually has significant hardware limitations and it is not an option to allow for the possibility of memory errors: heap, stack
Compiler certification - while verifying the correctness of the application code is one thing, the method of proving the object code generated by the compiler is a whole new bag of worms.
Can anyone help me address these concerns?
Edit: Thanks for the feedback everyone! I have a lot of resources to research, so far:
I've looked at everything at a high level and the resources: COMPCERT and Atom address the issues above and Copilot seems to add nice functionality on top of Atom. I still have to figure out if/what can work together and what projects such as SMACCM add to the above technologies.submitted by Azsu
[link] [21 comments]