News aggregator

Douglas M. Auclair (geophf): Dylan: the harsh realities of the market

Planet Haskell - Thu, 08/28/2014 - 10:06am
So, this is a little case study.

I did everything for Dylan. And when I say everything, I mean everything.  Here's my resumé:

  • I got excited about Dylan as a user, and I used it. I bought an old Mac that I don't ever remember the designation for, it's so '90's old, and got the floppies for the Dylan IDE from Apple research.
I'm not joking.
  • I integrated Dylan into my work at work, building an XML parser then open-sourcing it to the community under the (then) non-restrictive license. I think mine was the only XML parser that was industrial-strength for Dylan. Can't claim originality: I ported over the Common-LISP one, but it was a lot of (fun) work.
  • I made improvements to the gwydion-dylan compiler, including some library documentation (you can see my name right there, right in the compiler code), including some library functionality, did I work on the compiler itself? The Dylan syntax extensions or type system? I don't recall; if not in those places, I know I've looked at those guts: I had my fingers all over parts of the compiler.
I was in the Dylan compiler code. For you ll-types ('little language') that's no big deal.
But ask a software developer in industry if they've ever been in their compiler code. I have, too: I've found bugs in Java Sun-compiler that I fixed locally and reported up the chain.
  • I taught a course at our community college on Dylan. I had five students from our company that made satellite mission software.
  • I effing had commercial licenses bought when the boss asked me: what do we have to do to get this (my system) done/integrated into the build. I put my job on the line, for Dylan. ... The boss bought the licenses: he'd rather spend the $x than spending six weeks to back-port down to Java or C++.
  • I built a rule-based man-power scheduling system that had previously took three administrative assistants three days each quarter to generate. My system did it, and printed out a PDF in less than one second. I sold it, so that means I started a commercial company and sold my software.
I sold commercial Dylan software. That I wrote. Myself. And sold. Because people bought it. Because it was that good.
Hells yeah.
Question: what more could I have done?
I kept Dylan alive for awhile. In industry. For real.
So why is Dylan dead?
That's not the question.
Or, that question is answered over and over and over again.
Good languages, beautiful languages, right-thing languages languish and die all the time.
Dylan was the right-thing, and they (Apple) killed it in the lab, and for a reason.
Who is Dylan for?
That's not the question either. Because you get vague, general, useless answers.
The question is to ask it like Paul Graham answered it for LISP.
Lisp is a pointless, useless, weird language that nobody uses.
But Paul and his partner didn't care. They didn't give a ...
... what anybody else thought. They knew that this language, the language they loved, was built and designed and made for them. Just them and only them, because the only other people who were using it were college kids on comp.lang.lisp asking for the answers for problem-set 3 on last night's homework.
That's what Lisp was good for: nothing.That's who Lisp was good for: nobody.
Same exact scenario for Erlang. Exactly the same. Erlang was only good for Joe Armstrong and a couple of buddies/weirdos like him, you know: kooks, who believed that Erlang was the right-thing for what they were doing, because they were on a mission, see, and nothing nobody could say could stop them nor stand against them, and all who would rise up against them would fall.
What made Lisp and Haskell and Erlang and Scala and Prolog (yes, Prolog, although you'll never hear that success story publicly, but $26M and three lives saved? Because of a Prolog system I wrote? And that's just one day in one month's report for data? I call that a success) work when nobody sane would say that these things would work?
Well, it took a few crazy ones to say, no, not: 'say' that it would work, but would make it work with their beloved programming language come hell or high water or, worse: indifferent silence, or ridicule, or pity from the rest of the world.
That is the lesson of perl and python and all these other languages. They're not good for anything. They suck. And they suck in libraries and syntax and semantics and weirdness-factor and everything.
But two, not one, but at least two people loved that language enough to risk everything, and ...
They lost.
Wait. What?
Did you think I was going to paint the rosy picture and lie to you and say 'they won'?
Because they didn't.
Who uses Lisp commercially? Or Haskell, except some fringers, or Scala or Clojure or Erlang or Smalltalk or Prolog
... or Dylan.
These languages are defined, right there in the dictionary.
Erlang: see 'career wrecker.'
Nobody uses those languages nor admits to even touching them with a 10-foot (3-meter) pole. I had an intern from college. 'Yeah, we studied this weird language called ML in Comp.sci. Nobody uses it.'
She was blown away when I started singing ML's praises and what it can do.
A meta-language, and she called it useless? Seriously?
Because that's what the mainstream sees.
Newsflash. I'm sorry. Dylan, Haskell, Idris: these aren't main-stream, and they never will be.
Algebraic types? Dependent types? You'll never see them. They're too ... research-y. They stink of academe, which is: they stink of uselessness-to-industry. You'll be dead and buried to see them in this form, even after they discover the eternity elixir. Sorry.
Or you'll see them in Visual Basic as a new Type-class form that only a few Microserfs use because they happened to have written those extensions. Everybody else?
Here's how Dylan will succeed, right now.
Bruce and I will put our heads together, start a company, and we'll code something. Not for anybody else to use and to love and to cherish, just for us, only for us, and it will blow out the effing doors, and we'll be bought out for $40M because our real worth is $127M.
And the first thing that Apple will do, after they bought us, is to show us the door, then convert the code into Java. Or Swift. Or Objective-C, or whatever.
And that's how we'll win.
Not the $40M. Not the lecture series on 'How to Make Functional Programming Work in Industry for Real' afterwards at FLoC and ICFP conferences with fan-bois and -girls wanting to talk to us afterwards and ask us how they can get a job doing functional programming.
Not that.
We'll win because we made something in Dylan, and it was real, and it worked, and it actually did something for enough people that we can now go to our graves knowing that we did something once with our lives (and we can do it again and again, too: there's no upper limit on the successes you're allowed to have, people) that meant something to some bodies. And we did that. With Dylan.
I've done that several times already, by my counting: the Prolog project, the Dylan project, the Mercury project, and my writing.
I'm ready to do that, again.
Because, actually, fundamentally, doing something in this world and for it ... there's nothing like it.
You write that research paper, and I come up to you, waving it in your face, demanding you implement your research because I need it to do my job in Industry?
I've done that to three professors so far. Effing changed their world-view in that moment. "What?" they said, to a person, "somebody actually wants to use this?" The look of bemused surprise on their faces?

It was sad, actually, because they did write something that somebody out there (moiself) needed, but they never knew that what they were doing meant something.

And it did.
Effing change your world-view. Your job? Your research? Your programming language?
That's status quo, and that's good and necessary and dulce and de leche (or decorum, I forget which).
But get up out of the level you're at, and do something with it so that that other person, slouched in their chair, sits up and takes notice, and a light comes over their face and they say, 'Ooh! That does that? Wow!' and watch their world change, because of you and what you've done.
Dylan is for nothing and for nobody.
So is everything under the Sun, my friend.
Put your hand to the plow, and with the sweat of your brow, make it yours for this specific thing.
Regardless of the long hours, long months of unrewarded work, and regardless of the hecklers, naysayers, and concerned friends and parents, and regardless of the mountain of unpaid bills.
You make it work, and you don't stop until it does.
That's how I've won.
Every time.
Categories: Offsite Blogs

Well-Typed.Com: Dealing with Asynchronous Exceptions during Resource Acquisition

Planet Haskell - Thu, 08/28/2014 - 9:48am

Consider the following code: we open a socket, compute with it, and finally close the socket again. The computation happens inside an exception handler (try), so even when an exception happens we still close the socket:

example1 :: (Socket -> IO a) -> IO a example1 compute = do -- WRONG s <- openSocket r <- try $ compute s closeSocket s case r of Left ex -> throwIO (ex :: SomeException) Right a -> return a

Although this code correctly deals with synchronous exceptions–exceptions that are the direct result of the execution of the program–it does not deal correctly with asynchronous exceptions–exceptions that are raised as the result of an external event, such as a signal from another thread. For example, in

example2 :: (Socket -> IO a) -> IO (Maybe a) example2 compute = timeout someTimeout $ example1 compute

it is possible that the timeout signal arrives after we have opened the socket but before we have installed the exception handler (or indeed, after we leave the scope of the exception handler but before we close the socket). In order to address this we have to control precisely where asynchronous exceptions can and cannot be delivered:

example3 :: (Socket -> IO a) -> IO a example3 compute = mask $ \restore -> do s <- openSocket r <- try $ restore $ compute s closeSocket s case r of Left ex -> throwIO (ex :: SomeException) Right a -> return a

We mask asynchronous exceptions, and then restore them only inside the scope of the exception handler. This very common pattern is captured by the higher level combinator bracket, and we might rewrite the example as

example4 :: (Socket -> IO a) -> IO a example4 = bracket openSocket closeSocket Allowing asynchronous exceptions during resource acquisition

Suppose that we wanted to define a derived operation that opens a socket and performs some kind of handshake with the server on the other end:

openHandshake :: IO Socket openHandshake = do mask $ \restore -> do s <- openSocket r <- try $ restore $ handshake s case r of Left ex -> closeSocket s >> throwIO (ex :: SomeException) Right () -> return s

(These and the other examples can be defined in terms of bracket and similar, but we use mask directly so that it’s easier to see what is happening.) We might use openHandshake as follows:

example5 :: (Socket -> IO a) -> IO a example5 compute = do mask $ \restore -> do s <- openHandshake r <- try $ restore $ compute s closeSocket s case r of Left ex -> throwIO (ex :: SomeException) Right a -> return a

There are no resource leaks in this code, but there is a different problem: we call openHandshake with asynchronous exceptions masked. Although openHandshake calls restore before doing the handshake, restore restores the masking state to that of the enclosing context. Hence the handshake with the server cannot be timed out. This may not be what we want–we may want to be able to interrupt example5 with a timeout either during the handshake or during the argument computation.

Note that this is not a solution:

example6 :: (Socket -> IO a) -> IO a example6 compute = do mask $ \restore -> do s <- restore openHandshake -- WRONG r <- try $ restore $ compute s closeSocket s case r of Left ex -> throwIO (ex :: SomeException) Right a -> return a

Consider what might happen: if an asynchronous exception is raised after openHandshake returns the socket, but before we leave the scope of restore, the asynchronous exception will be raised and the socket will be leaked. Installing an exception handler does not help: since we don’t have a handle on the socket, we cannot release it.

Interruptible operations

Consider this definition from the standard libraries:

withMVar :: MVar a -> (a -> IO b) -> IO b withMVar m io = mask $ \restore -> do a <- takeMVar m b <- restore (io a) `onException` putMVar m a putMVar m a return b

This follows almost exactly the same pattern as the examples we have seen so far; we mask asynchronous exceptions, take the contents of the MVar, and then execute some operation io with the contents of the MVar, finally putting the contents of the MVar back when the computation completes or when an exception is raised.

An MVar acts as a lock, with takeMVar taking the role of acquiring the lock. This may, of course, take a long time if the lock is currently held by another thread. But we call takeMVar with asynchronous exceptions masked. Does this mean that the takeMVar cannot be timed out? No, it does not: takeMVar is a so-called interruptible operation. From the Asynchronous Exceptions in Haskell paper:

Any operation which may need to wait indefinitely for a resource (e.g., takeMVar) may receive asynchronous exceptions even within an enclosing block, but only while the resource is unavailable. Such operations are termed interruptible operations. (..) takeMVar behaves atomatically when enclosed in block. The takeMVar may receive asynchronous exceptions right up until the point when it acquires the MVar, but not after.

(block has been replaced by mask since the publication of the paper, but the principle is the same.) Although the existence of interruptible operations makes understanding the semantics of mask harder, they are necessary: like in the previous section, wrapping takeMVar in restore is not safe. If we really want to mask asynchronous exceptions, even across interruptible operations, Control.Exception offers uninterruptibleMask.

Custom interruptible operations

So an interruptible operation is one that can be interrupted by an asynchronous exception even when asynchronous exceptions are masked. Can we define our own interruptible operations? Yes, we can:

-- | Open a socket and perform handshake with the server -- -- Note: this is an interruptible operation. openHandshake' :: IO Socket openHandshake' = mask_ $ do s <- openSocket r <- try $ unsafeUnmask $ handshake s case r of Left ex -> closeSocket s >> throwIO (ex :: SomeException) Right () -> return s

unsafeUnmask is defined in GHC.IO, and unmasks asynchronous exceptions, no matter what the enclosing context is. This is of course somewhat dangerous, because now calling openHandshake' inside a mask suddenly opens up the possibility of an asynchronous exception being raised; and the only way to know is to look at the implementation of openHandshake', or its Haddock documentation. This is somewhat unsatisfactory, but exactly the same goes for takeMVar and any other interruptible operation, or any combinator that uses an interruptible operation under the hood. A sad state of affairs, perhaps, but one that we don’t currently have a better solution for.

Actually, using unsafeUnmask is a bit too crude. Control.Exception does not export it, but does export

allowInterrupt :: IO () allowInterrupt = unsafeUnmask $ return ()

with documentation

When invoked inside mask, this function allows a blocked asynchronous exception to be raised, if one exists. It is equivalent to performing an interruptible operation, but does not involve any actual blocking.

When called outside mask, or inside uninterruptibleMask, this function has no effect.

(emphasis mine.) Sadly, this documentation does not reflect the actual semantics: unsafeUnmask, and as a consequence allowInterrupt, unmasks asynchronous exceptions no matter what the enclosing context is: even inside uninterruptibleMask. We can however define our own operator to do this:

interruptible :: IO a -> IO a interruptible act = do st <- getMaskingState case st of Unmasked -> act MaskedInterruptible -> unsafeUnmask act MaskedUninterruptible -> act

where we call unsafeUnmask only if the enclosing context is mask, but not if it is uninterruptibleMask (TODO: What is the semantics when we nest these two?). We can use it as follows to define a better version of openHandshake:

-- | Open a socket and perform handshake with the server -- -- Note: this is an interruptible operation. openHandshake' :: IO Socket openHandshake' = mask_ $ do s <- openSocket r <- try $ interruptible $ handshake s case r of Left ex -> closeSocket s >> throwIO (ex :: SomeException) Right () -> return s Resource allocation timeout

If we wanted to timeout the allocation of the resource only, we might do

example7 :: (Socket -> IO a) -> IO a example7 compute = do mask $ \restore -> do ms <- timeout someTimeout $ openHandshake' case ms of Nothing -> throwIO (userError "Server busy") Just s -> do r <- try $ restore $ compute s closeSocket s case r of Left ex -> throwIO (ex :: SomeException) Right a -> return a

Exceptions are masked when we enter the scope of the timeout, and are unmasked only once we are inside the exception handler in openHandshake'–in other words, if a timeout happens, we are guaranteed to clean up the socket. The surrounding mask is however necessary. For example, suppose we are writing some unit tests and we are testing openHandshake'. This is wrong:

example8 :: IO () example8 = do ms <- timeout someTimeout $ openHandshake' case ms of Just s -> closeSocket s Nothing -> return ()

Even if we are sure that the example8 will not be interrupted by asynchronous exceptions, there is still a potential resource leak here: the timeout exception might be raised just after we leave the mask_ scope from openHandshake but just before we leave the timeout scope. If we are sure we don’t need to worry about other asynchronous exceptions we can write

example8 :: IO () example8 = do s <- mask_ $ timeout someTimeout $ openHandshake' case ms of Just s -> closeSocket s Nothing -> return ()

although of course it might be better to simply write

example8 :: IO () example8 = bracket (timeout someTimeout $ openHandshake') (\ms -> case ms of Just s -> closeSocket s Nothing -> return ()) (\_ -> return ()) Conclusions

Making sure that resources are properly deallocated in the presence of asynchronous exceptions is difficult. It is very important to make sure that asynchronous exceptions are masked at crucial points; unmasking them at the point of calling a resource allocation function is not safe. If you nevertheless want to be able to timeout resource allocation, you need to make your resource allocation function interruptible.

For completeness’ sake, there are some other solutions that avoid the use of unsafeUnmask. One option is to thread the restore argument through (and compose multiple restore arguments if there are multiple nested calls to mask). This requires resource allocations to have a different signature, however, and it is very error prone: a single mask somewhere along the call chain where we forget to thread through the restore argument will mean the code is no longer interruptible. The other option is to run the code that we want to be interruptible in a separate thread, and wait for the thread to finish with, for example, a takeMVar. Getting this right is however no easy task, and it doesn’t change anything fundamentally anyway: rather than using unsafeUnmask we are now using a primitive interruptible operation; either way we introduce the possibility of exceptions even in the scope of mask_.

Finally, when your application does not fit the bracket pattern we have been using (implicitly or explicitly), you may want to have a look at resourcet and pipes or conduit, or my talk Lazy I/O and Alternatives in Haskell.

Categories: Offsite Blogs

Haskell Weekly News: Issue 303

General haskell list - Thu, 08/28/2014 - 6:40am
Welcome to issue 303 of the HWN, an issue covering crowd-sourced bits of information about Haskell from around the web. This issue covers from August 17 to 23, 2014 Quotes of the Week * monochrom: "point free" can be decomposed to: "point" refers to ".", "free" refers to using no "$". :) Top Reddit Stories * λ Bubble Pop! Domain:, Score: 97, Comments: 41 Original: [1] On Reddit: [2] * The fundamental problem of programming language package management Domain:, Score: 82, Comments: 54 Original: [3] On Reddit: [4] * How Programming language subreddits talk (including Haskell) Domain:, Score: 72, Comments: 31 Original: [5] On Reddit: [6] * A fast, generic and type-safe image processing library written in Haskell Domain:, Score: 59, Comm
Categories: Incoming News

haskell on hadoop

Haskell on Reddit - Thu, 08/28/2014 - 5:48am

has anyone used haskell on hadoop? What were your experiences? I found hadron which looks pretty nice. (Here's a nice talk on hadron that was posted here a while back.)

Edit: I also need hbase bindings, but it looks like hbase-haskell hasn't been touched for a while

submitted by ludflu
[link] [9 comments]
Categories: Incoming News

Would it be plausible to script games in Haskell on top of a C/C++ engine?

Haskell on Reddit - Thu, 08/28/2014 - 2:04am

Just a random thought of mine I wanted to get an opinion on. I currently C and Python for video game programming but my favorite language is Haskell. Would something like this work well?

submitted by ProbablyALinuxUser
[link] [35 comments]
Categories: Incoming News

Keegan McAllister: Calling a Rust library from C (or anything else!)

Planet Haskell - Thu, 08/28/2014 - 12:18am

One reason I'm excited about Rust is that I can compile Rust code to a simple native-code library, without heavy runtime dependencies, and then call it from any language. Imagine writing performance-critical extensions for Python, Ruby, or Node in a safe, pleasant language that has static lifetime checking, pattern matching, a real macro system, and other goodies like that. For this reason, when I started html5ever some six months ago, I wanted it to be more than another "Foo for BarLang" project. I want it to be the HTML parser of choice, for a wide variety of applications in any language.

Today I started work in earnest on the C API for html5ever. In only a few hours I had a working demo. And this is a fairly complicated library, with 5,000+ lines of code incorporating

It's pretty cool that we can use all this machinery from C, or any language that can call C. I'll describe first how to build and use the library, and then I'll talk about the implementation of the C API.

html5ever (for C or for Rust) is not finished yet, but if you're feeling adventurous, you are welcome to try it out! And I'd love to have more contributors. Let me know on GitHub about any issues you run into.

Using html5ever from C

Like most Rust libraries, html5ever builds with Cargo.

$ git clone
$ cd html5ever
$ git checkout dev
$ cargo build
Updating git repository ``
Compiling phf_mac v0.0.0 (
Compiling html5ever-macros v0.0.0 (file:///tmp/html5ever)
Compiling phf v0.0.0 (
Compiling html5ever v0.0.0 (file:///tmp/html5ever)

The C API isn't Cargo-ified yet, so we'll build it using the older Makefile-based system.

$ mkdir build
$ cd build
$ ../configure
$ make libhtml5ever_for_c.a
rustc -D warnings -C rpath -L /tmp/html5ever/target -L /tmp/html5ever/target/deps \
-o libhtml5ever_for_c.a --cfg for_c --crate-type staticlib /tmp/html5ever/src/
warning: link against the following native artifacts when linking against this static library
note: the order and any duplication can be significant on some platforms, and so may need to be preserved
note: library: rt
note: library: dl
note: library: pthread
note: library: gcc_s
note: library: pthread
note: library: c
note: library: m

Now we can build an example C program using that library, and following the link instructions produced by rustc.

$ H5E_PATH=/tmp/html5ever
$ gcc -Wall -o tokenize tokenize.c -I $H5E_PATH/capi -L $H5E_PATH/build \
-lhtml5ever_for_c -lrt -ldl -lpthread -lgcc_s -lpthread -lc -lm

$ ./tokenize 'Hello&comma; <i class=excellent>world!</i>'
CHARS : Hello
TAG : <i>
ATTR: class="excellent"
CHARS : world!
TAG : </i>

The build process is pretty standard for C; we just link a .a file and its dependencies. The biggest obstacle right now is that you won't find the Rust compiler in your distro's package manager, because the language is still changing so rapidly. But there's a ton of effort going into stabilizing the language for a Rust 1.0 release this year. It won't be too long before rustc is a reasonable build dependency.

Let's look at the C client code.

#include <stdio.h>

#include "html5ever.h"

void put_str(const char *x) {
fputs(x, stdout);

void put_buf(struct h5e_buf text) {
fwrite(, text.len, 1, stdout);

void do_start_tag(void *user, struct h5e_buf name, int self_closing, size_t num_attrs) {
put_str("TAG : <");
if (self_closing) {

// ...

struct h5e_token_ops ops = {
.do_chars = do_chars,
.do_start_tag = do_start_tag,
.do_tag_attr = do_tag_attr,
.do_end_tag = do_end_tag,

struct h5e_token_sink sink = {
.ops = &ops,
.user = NULL,

int main(int argc, char *argv[]) {
if (argc < 2) {
printf("Usage: %s 'HTML fragment'\n", argv[0]);
return 1;

struct h5e_tokenizer *tok = h5e_tokenizer_new(&sink);
h5e_tokenizer_feed(tok, h5e_buf_from_cstr(argv[1]));
return 0;

The struct h5e_token_ops contains pointers to callbacks. Any events we don't care to handle are left as NULL function pointers. Inside main, we create a tokenizer and feed it a string. html5ever for C uses a simple pointer+length representation of buffers, which is this struct h5e_buf you see being passed by value.

This demo only does tokenization, not tree construction. html5ever can perform both phases of parsing, but the API surface for tree construction is much larger and I didn't get around to writing C bindings yet.

Implementing the C API

Some parts of Rust's libstd depend on runtime services, such as task-local data, that a C program may not have initialized. So the first step in building a C API was to eliminate all std:: imports. This isn't nearly as bad as it sounds, because large parts of libstd are just re-exports from other libraries like libcore that we can use with no trouble. To be fair, I did write html5ever with the goal of a C API in mind, and I avoided features like threading that would be difficult to integrate. So your library might give you more trouble, depending on which Rust features you use.

The next step was to add the #![no_std] crate attribute. This means we no longer import the standard prelude into every module. To compensate, I added use core::prelude::*; to most of my modules. This brings in the parts of the prelude that can be used without runtime system support. I also added many imports for ubiquitous types like String and Vec, which come from libcollections.

After that I had to get rid of the last references to libstd. The biggest obstacle here involved macros and deriving, which would produce references to names under std::. To work around this, I create a fake little mod std which re-exports the necessary parts of core and collections. This is similar to libstd's "curious inner-module".

I also had to remove all uses of format!(), println!(), etc., or move them inside #[cfg(not(for_c))]. I needed to copy in the vec!() macro which is only provided by libstd, even though the Vec type is provided by libcollections. And I had to omit debug log messages when building for C; I did this with conditionally-defined macros.

With all this preliminary work done, it was time to write the C bindings. Here's how the struct of function pointers looks on the Rust side:

pub struct h5e_token_ops {
do_start_tag: extern "C" fn(user: *mut c_void, name: h5e_buf,
self_closing: c_int, num_attrs: size_t),

do_tag_attr: extern "C" fn(user: *mut c_void, name: h5e_buf,
value: h5e_buf),

do_end_tag: extern "C" fn(user: *mut c_void, name: h5e_buf),

// ...

The processing of tokens is straightforward. We pattern-match and then call the appropriate function pointer, unless that pointer is NULL. (Edit: eddyb points out that storing NULL as an extern "C" fn is undefined behavior. Better to use Option<extern "C" fn ...>, which will optimize to the same one-word representation.)

To create a tokenizer, we heap-allocate the Rust data structure in a Box, and then transmute that to a raw C pointer. When the C client calls h5e_tokenizer_free, we transmute this pointer back to a box and drop it, which will invoke destructors and finally free the memory.

You'll note that the functions exported to C have several special annotations:

  • #[no_mangle]: skip name mangling, so we end up with a linker symbol named h5e_tokenizer_free instead of _ZN5for_c9tokenizer18h5e_tokenizer_free.
  • unsafe: don't let Rust code call these functions unless it promises to be careful.
  • extern "C": make sure the exported function has a C-compatible ABI. The data structures similarly get a #[repr(C)] attribute.

Then I wrote a C header file matching this ABI:

struct h5e_buf {
unsigned char *data;
size_t len;

struct h5e_buf h5e_buf_from_cstr(const char *str);

struct h5e_token_ops {
void (*do_start_tag)(void *user, struct h5e_buf name,
int self_closing, size_t num_attrs);

void (*do_tag_attr)(void *user, struct h5e_buf name,
struct h5e_buf value);

void (*do_end_tag)(void *user, struct h5e_buf name);

/// ...

struct h5e_tokenizer;

struct h5e_tokenizer *h5e_tokenizer_new(struct h5e_token_sink *sink);
void h5e_tokenizer_free(struct h5e_tokenizer *tok);
void h5e_tokenizer_feed(struct h5e_tokenizer *tok, struct h5e_buf buf);
void h5e_tokenizer_end(struct h5e_tokenizer *tok);

One remaining issue is that Rust is hard-wired to use jemalloc, so linking html5ever will bring that in alongside the system's libc malloc. Having two separate malloc heaps will likely increase memory consumption, and it prevents us from doing fun things like allocating Boxes in Rust that can be used and freed in C. Before Rust can really be a great choice for writing C libraries, we need a better solution for integrating the allocators.

If you'd like to talk about calling Rust from C, you can find me as kmc in #rust and #rust-internals on And if you run into any issues with html5ever, do let me know, preferably by opening an issue on GitHub. Happy hacking!

Categories: Offsite Blogs

Haskell Platform 2014 build problems on OS X

haskell-cafe - Wed, 08/27/2014 - 9:39pm
Hello! I just want to point to your attention this issue of Homebrew: It seems that they’re forced to remove the formula for the Haskell Platform because, it seems, the build process has become extremely complicated and they can’t figure out how to automate it. Is there someone that know enough internals of the platform to help them or to give some advice? Thanks, Nicola
Categories: Offsite Discussion

conduit stream fusion

Haskell on Reddit - Wed, 08/27/2014 - 8:04pm
Categories: Incoming News

ANN: cgi 3001.2.0.0

General haskell list - Wed, 08/27/2014 - 7:32pm
Hello, I've just uploaded a new version of cgi to hackage [1]. This release supports GHC 7.8.3, 7.6.3, and 7.4.2. I'm also looking for co-maintainers or backup maintainers of cgi, please email me directly if you're interested. This release was made possible with the assistance and guidance of Carter Schonwald and was based on a patch by Alexander Vershilov. Thanks. [1]
Categories: Incoming News

I'm a .NET developer. What can Haskell/Functional Programming teach me?

Haskell on Reddit - Wed, 08/27/2014 - 7:20pm

Hi all, I am a (primarily) .NET developer, though experienced in Smalltalk, Ruby, Java, and a whole list of other things. I like to keep a wide horizon and I've gathered that Functional Programming can teach me good things. A lot of what I do day to day involves the web, typically APIs on HTTP endpoints. I'm not about to completely jump ship to Haskell and FP, but I am certain they can teach me a thing or two and I am eager to embrace it.

I have been using F# a lot lately, but as it is "yet another" .NET language it doesn't feel like true FP. Well, it can be, but the availability of libraries from other languages that clearly aren't FP makes it just too easy to break away from FP I guess? Anyway, I digress.

I understand the syntax of Haskell just fine, I've done plenty of stuff in the GHCI and read many a tutorial. However it just isn't sinking in, I still feel like I am missing something. Like there is some beautiful unicorn of simplicity that is just waiting for me to find it/unlearn the way OO does things/etc.

So, my understanding is that Haskell is a "true" Functional Programming language/platform. I've installed Haskell Platform, and EclipseFP (I'm on Windows without access to *nix for now) and I'm just staring at my project workspace still wondering what I'm not seeing. I've thus far compiled a small web-service using Scotty; However Scotty too didn't feel like FP should feel to me. Shoe-horning the way Sinatra does it into Haskell feels odd.

Am I right to feel odd about this? What am I missing that I should be learning? What would you suggest as a project/task/whatever for me to "get" Functional Programming (and/or Haskell)?

The "No Side Effects" functions "rule": What rule of thumb percentage of the application should this be? Is 100% no side effects even possible? Even with persistence/sessions/cookies/etc?

Many thanks in advance, and I'm sure this question(s) is asked many times so I apologise for that.

submitted by Jestar342
[link] [31 comments]
Categories: Incoming News

Conduit+GHC high memory use for simple Sink

haskell-cafe - Wed, 08/27/2014 - 7:19pm
Hello Cafe, First I'd like to thank Michael Snoyman and Gabriel Gonzalez for the work they've done on the conduit and pipes stream processing libraries, and all the accompanying tutorial content. I've been having fun converting a text processing app from C to Haskell. I'm seeing unexpectedly high memory usage in a stream-processing program that uses the conduit library. I've created a example that demonstrates the problem. The program accepts gzip files as arguments, and for each one, classifies each line as either Even or Odd depending on the length, then outputs some result depending on the Sink used. For each gzip file: action :: GzipFilePath -> IO () action (GzipFilePath filePath) = do result <- runResourceT $ CB.sourceFile filePath $$ Zlib.ungzip =$ CB.lines =$ token =$ sink2 putStrLn $ show result The problem is the following Sink, which counts how many even/odd Tokens are se
Categories: Offsite Discussion

Can somebody explain me the instance Generic Int

Haskell on Reddit - Wed, 08/27/2014 - 3:26pm

Quoting from

-- Int data D_Int data C_Int instance Datatype D_Int where datatypeName _ = "Int" moduleName _ = "GHC.Int" instance Constructor C_Int where conName _ = "" -- JPM: I'm not sure this is the right implementation... instance Generic Int where type Rep Int = D1 D_Int (C1 C_Int (S1 NoSelector (Rec0 Int))) from x = M1 (M1 (M1 (K1 x))) to (M1 (M1 (M1 (K1 x)))) = x

Can somebody explain me what the idea between the three nested M1s is?

And why is the instance Generic Integer missing?

submitted by nh2_
[link] [8 comments]
Categories: Incoming News

[ANN] Wheb 0.3 release

haskell-cafe - Wed, 08/27/2014 - 2:26pm
Hello all, I have been working on a web framework that I use for my personal projects and I thought I would share my small progress. I started this project several months ago when I hit a big bump in difficulty going from a simple skeleton project to a more intricate project using alternative frameworks. Lately I've added three main features: WebSockets, Cache and Redis. WebSockets use the same WhebT interface as the standard HTTP handlers allowing you to share functions and middlware accross your application. Cache is a new plugin for simplistic Caching. It provides an interface to cache ByteStrings under Text keys. Like the other plugins, it is backend agnostic so you can write your own backend or use the prebuilt Redis cache (this requires you to configure your redis instance for caching). In addition to Redis acting as a Cache, you can also use it as a database and as a backend for the plugins Auth and Sessions. You can mix Redis with other plugins. For one of my projects I use the MongoDB plugin f
Categories: Offsite Discussion

Philip Wadler: Informatics Independence Referendum Debate - the result

Planet Haskell - Wed, 08/27/2014 - 7:43am

Here is the outcome of the debate announced earlier:
<embed allowfullscreen="true" allowscriptaccess="always" flashvars="config=;file=" height="360" src="" width="740"></embed>
Yes 19
No 25
Undecided 28

Yes 28
No 31
Undecided 26

(Either some people entered after the debate began, or some people began the debate unsure even whether they were undecided.)

Thank you to Alan, Mike, and the audience for a fantastic debate. The audience asked amazing questions on both sides, clearly much involved with the process.

Video of debate here.
Alan's slides here.
My slides here.

Categories: Offsite Blogs

Ivory Language

Haskell on Reddit - Wed, 08/27/2014 - 12:54am
Categories: Incoming News

A Face Only A Mother Could Love.

Haskell on Reddit - Tue, 08/26/2014 - 8:44pm
Categories: Incoming News

Simon Michael: Creating well-behaved Hakyll blog posts

Planet Haskell - Tue, 08/26/2014 - 8:15pm

Posts in a Hakyll-powered blog need to be created with care, if you want your feed to work well with clients and aggregators. There are many things to remember:

  • If you have clones of your site, decide which one to work in and make sure it’s up to date
  • Create the file in the right place
  • Name it consistently (I use
  • In my setup, prefix it with _ if it’s a draft (I render but don’t publish those)
  • Set title, tags, and author with a metadata block
  • Set published time with metadata to get a more precise timestamp than Hakyll can guess from the filename. Include a time zone. Use the right format.
  • When moving a post from draft to published:
    • Update the published time
    • Update the file name if the title or publish date has changed
  • If changing a post after it has been published: set updated time in the metadata
  • At some point, commit it to version control and sync it to other clones

I find this makes blogging feel tedious, especially after an absence when I’ve forgotten the details. Case in point, I managed to share an ugly template post with Planet Haskell readers while working on this one.

So I’m trying out this bash shell script, maybe it will help. Adjust to suit your setup.
(updated 2014/8/27)

# add to ~/.bashrc BLOGDIR=~/src/ # List recent blog posts. alias blog-ls="ls $BLOGDIR | tail -10" # Create a new hakyll-compatible draft blog post. # blog-new ["The Title" ["tag1, tag2" ["Author Name"]]] function blog-new() { ( TITLE=${1:-Default Title} TAGS=${2:-defaulttag1, defaulttag2} AUTHOR=${3:-Default Author Name} SLUG=${TITLE// /-} DATE=`date +"%Y-%m-%d"` TIME=`date +"%Y-%m-%d %H:%M:%S%Z"` FILE=_$DATE-$ echo creating $BLOGDIR/$FILE cat <<EOF >>$BLOGDIR/$FILE && emacsclient -n $BLOGDIR/$FILE --- title: $TITLE tags: $TAGS author: $AUTHOR published: $TIME --- EOF ) }

An example:

$ blog-new 'Scripted Hakyll blog post creation' 'hakyll, haskell' creating (file opens in emacs, edit & save) $ make ./site build Initialising... Creating store... Creating provider... Running rules... Checking for out-of-date items Compiling updated blog/ Success

See also part 2.

Categories: Offsite Blogs

Simon Michael: Well-behaved Hakyll blog posts, continued

Planet Haskell - Tue, 08/26/2014 - 8:00pm

More hakyll blog fixes:

Ugly things showing on planets

My posts were showing unwanted things on planet haskell - double heading, redundant date, tag links, and ugly disqus html. By comparing with Jasper Van der Jeugt’s blog, I found the problem: I was snapshotting content for the feed at the wrong time, after applying the post template:

>>= return . fmap demoteHeaders >>= loadAndApplyTemplate "templates/post.html" (postCtx tags) >>= saveSnapshot "content" >>= loadAndApplyTemplate "templates/default.html" defaultContext


>>= saveSnapshot "content" -- >>= return . fmap demoteHeaders >>= loadAndApplyTemplate "templates/post.html" (postCtx tags) >>= loadAndApplyTemplate "templates/default.html" defaultContext

Manual feed publishing

The main blog feed is now generated with a _ prefix, and I must manually rename it (with make feed) to make it live it on Planet Haskell. This will hopefully reduce snafus (and not create new ones).

./site.hs 95 - create ["blog.xml"] $ do + create ["_blog.xml"] $ do ./Makefile 14 +feed: _site/blog.xml + +_site/blog.xml: _site/_blog.xml + cp _site/_blog.xml _site/blog.xml +

Better HTML titles

Changed the “Joyful Systems” prefix to a suffix in the HTML page titles, making search results and browser tab names more useful.

Categories: Offsite Blogs