News aggregator

So I re-implemented Freenet in Haskell, what now?

Haskell on Reddit - Thu, 05/01/2014 - 2:53pm

I've been trying Freenet on-and-off (well, mostly off) for over a decade now. I like the technical foundations it's built upon, but I always felt the implementation and presentation are lacking.

Some day I decided to have a look at it's code, maybe I can help out. That's what I thought... I don't want to offend anyone, so let's just say I did not like what I see.

Just to make a point I decided to re-implement (a subset of) what Freenet is in Haskell. I'm not good at names, so the result is called A Distributed Store. I am pleased say it works better than I expected. (please read the readme on GitHub for some details) But I feel like I've lost track right now.

Until recently it always was obvious to me what to do next, so I just did it. Now I'm out of low-hanging fruits and could use some feedback. There's still lots of stuff to do:

  • simply connecting to every node you possibly can is not a problem at the current network size of ~20, but it won't be good if the network will ever grow
  • I'd love to have a bundled implementation of FMS (Freenet Message System, a spam-free, pseudonymous forum system built on top of Freenet)
  • Proper support for up/downloading large files might be attractive for some audiences as well
  • Someone knowledgeable in crypto stuff might do a review, better before people actually start to bet their ass on this. But then, I've never heard of someone being busted for stuff he's been doing on Freenet. Maybe because nobody is doing anything on Freenet. Avoid success at all cost, you know? :-)
  • Someone knowledgeable in Haskell might point out the antipatterns I'm applying.

So, may I beg for some feedback dear Redditors? Or maybe someone hears his/her inner Cypherpunk and wants to jump on hacking on this? Yeah, that would be great...

(this is my first reddit post, so please pardon whatever I'm doing wrong)

submitted by waldheinz
[link] [34 comments]
Categories: Incoming News

Packaging Haskell Platform and GHC API?

Haskell on Reddit - Thu, 05/01/2014 - 1:48pm

Hey there,

I'd like to package the Haskell Platform and GHC (for GHC API) together in one bundle in order to distribute it with a Mac app. (Specifically, I am working on a Mac app for IHaskell, so beginners can download that to immediately get started playing with Haskell).

Does anyone have any experience with this? What's the easiest way to do this? My list of dependencies is fairly long:

  • Haskell Platform (well, all packages in it)
  • A few other packages installed via cabal
  • A native library (libzmq); cabal packages depend on it
  • Functioning Python > 2.6ish
  • GHC API

My current best idea is to package this all in a Virtualbox VM. I need to run a server that my Mac app client can use, so mount shared folders in the VM so that the VM can read/write to disk and expose some ports from the VM. If this is the best solution, what Linux distro would you suggest to use in the VM? I'd want something very lightweight.

I've also considered using Docker somehow, as IHaskell is already packaged with a Dockerfile. However, I'm not sure how I'd package docker so that its all doable via a single Mac app install.

Ideas? Suggestions?

Thanks! I know this isn't directly related to Haskell, but I'm hoping someone here has experience packaging Haskell applications.

submitted by NiftyIon
[link] [3 comments]
Categories: Incoming News

Johan Tibell: Ekg 0.4 Released

Planet Haskell - Thu, 05/01/2014 - 6:28am

It's been over two years since the last major release of ekg. Ever since the first release I knew that there were a number of features I wanted to have in ekg that I didn't implement back then. This release adds most of them.

Integration with other monitoring systems

When I first wrote ekg I knew it only solved half of the program monitoring problem. Good monitoring requires two things

  1. a way to track what your program is doing, and
  2. a way to gather and persist that data in a central location.

The latter is neccesary because

  • you don't want to lose your data if your program crashes (i.e. ekg only stores metrics in memory),
  • you want to get an aggregate picture of your whole system over time, and
  • you want to define alarms that go off if some metric passes some threshold.

Ekg has always done (1), as it provides a way to define metrics and inspect their values e.g. using your web browser or curl.

Ekg could help you to do (2), as you could use the JSON API to sample metrics and then push them to an exiting monitoring solution, such as Graphite or Ganglia. However, it was never really convenient.

Today (2) will get much easier.

Statsd integration

Statsd is

A network daemon that ... listens for statistics, like counters and timers, sent over UDP and sends aggregates to one or more pluggable backend services (e.g., Graphite).

Statsd is quite popular and has both client and server implementations in multiple languages. It supports quite a few backends, such as Graphite, Ganglia, and a number of hosted monitoring services. It's also quite easy to install and configure (although many of the backends it uses are not.)

Ekg can now be integrated with statsd, using the ekg-statsd package. With a few lines you can have your metrics sent to a statsd:

main = do store <- newStore -- Register some metrics with the metric store: registerGcMetrics store -- Periodically flush metrics to statsd: forkStatsd defaultStatsdOptions store

ekg-statsd can be used either together with ekg, if you also want the web interface, or standalone if the dependencies pulled in by ekg are too heavyweight for your application or if you don't care about the web interface. ekg has been extended so it can share the Server's metric store with other parts of the application:

main = do handle <- forkServer "localhost" 8000 forkStatsd defaultStatsdOptions (serverMetricStore handle)

Once you set up statsd and e.g. Graphite, the above lines are enough to make your metrics show up in Graphite:

image

Integration with your monitoring systems

The ekg APIs have been re-organized and the package split such that it's much easier to write your own package to integrate with the monitoring system of your choice. The core API for tracking metrics has been split out from the ekg package into a new ekg-core package. Using this package, the ekg-statsd implementation could be written in a mere 121 lines.

While integrating with other systems was technically possible in the past, using the ekg JSON API, it was both inconvenient and wasted CPU cycles generating and parsing JSON. Now you can get an in-memory representation of the metrics at a given point in time using the System.Metrics.sampleAll function:

-- | Sample all metrics. Sampling is /not/ atomic in the sense that -- some metrics might have been mutated before they're sampled but -- after some other metrics have already been sampled. sampleAll :: Store -> IO Sample -- | A sample of some metrics. type Sample = HashMap Text Value -- | The value of a sampled metric. data Value = Counter !Int64 | Gauge !Int64 | Label !Text | Distribution !Stats

All that ekg-statsd does is to call sampleAll periodically and convert the returned Values to UDP packets that it sends to statsd.

Namespaced metrics

In a large system each component may want to contribute their own metrics to the set of metrics exposed by the program. For example, the Snap web server might want to track the number of requests served, the latency for each request, the number of requests that caused an internal server error, etc. To allow several components to register their own metrics without name clashes, ekg now supports namespaces.

Namespaces also makes it easier to navigate metrics in UIs. For example, Graphite gives you a tree-like navigation of metrics based on their namespaces.

In ekg dots in metric names are now interpreted as separating namespaces. For example, the default GC metric names now all start with "rts.gc.". Snap could for example prefix all its metric names with "snap.". While this doesn't make collisions impossible, it should make them much less likely.

If your library want to provide a set of metrics for the application, it should provide a function that looks like this:

registerFooMetrics :: Store -> IO ()

The function should call the various register functions in System.Metrics. It should also document which metrics it registers. See System.Metrics.registerGcMetrics for an example.

A new metric type for tracking distributions

It's often desirable to track the distribution of some event. For example, you might want to track the distribution of response times for your webapp, so you can get notified if things are slow all of a sudden and so you can try to optimize the latency.

The new Distribution metric lets you do that.

Every time an event occurs, simply call the add function:

add :: Distribution -> Double -> IO ()

The add function takes a value which could represent e.g. the number of milliseconds it took to serve a request.

When the distribution metric is later sampled you're given a value that summarizes the distribtuion by providing you with the mean, variance, min/max, and so on.

The implementation uses an online algorithm to track these statistics so it uses O(1) memory. The algorithm is also numerically stable so the statistics should be accurate even for long-running programs.

While it didn't make this release, in the future you can look forward to being able to track both quantiles and keep histrograms of the events. This will let you track e.g. the 95-percentile response time of your webapp.

Counters and gauges are always 64-bits

To keep ekg more efficient even on 32-bit platforms, counters and gauges were stored as Int values. However, if a counter is increased 10,000 times per second, which isn't unusual for a busy server, such a counter would wrap around in less than 2.5 days. Therefore all counters and gauges are now stored as 64-bit values. While this is technically a breaking change, it shouldn't affect the majority of users.

Improved performance and multi-core scaling

I received a report of contention in ekg when multiple cores were used. This prompted me to improve the scaling of all metrics types. The difference is quite dramatic on my heavy contention benchmark:

+RTS -N1 +RTS -N6 Before 1.998s 82.565s After 0.117s 0.247s

The benchmark updates a single counter concurrently in 100 threads, performing 100,000 increments per thread. It was run on a 6 core machine. The cause of the contention was atomicModifyIORef, which has been replaced by an atomic-increment instruction. There are some details on the GHC Trac.

In short, you shouldn't see contention issues anymore. If you, I still have some optimizations that I didn't apply because the implementation should already be fast enough.

Categories: Offsite Blogs

ekg 0.4 released

Haskell on Reddit - Thu, 05/01/2014 - 5:34am
Categories: Incoming News

Philip Wadler: Net Neutrality, R.I.P.

Planet Haskell - Thu, 05/01/2014 - 5:17am

Brian McFadden nails it in The Nib. If you want to do something about it, look to Fight for the Future.
Categories: Offsite Blogs

Looking for people who know about semiring geometry to help me with a project

Haskell on Reddit - Thu, 05/01/2014 - 1:48am

I've taken an interest in an area of math known by the name tropical geometry. It's sort of new, been around about ten years (as far as algebraic geometers are concerned).

Anyway, I digress. I've been trying to do research in the area, and it occurred to me that it would really help if I could efficiently perform tropical calculations, and make tropical graphs. So, I've been working on a Haskell library to do so. You can look at what I have so far on GitHub (it's not much).

I'm at the point where I would like to make certain major decisions about the project, but I don't have the foresight nor the expertise to do so, and I'm looking for someone who does.

Any insight, advice, criticism, what-have-you, would be greatly appreciated.

submitted by l1cust
[link] [9 comments]
Categories: Incoming News

LYAH solveRPN code query

Haskell on Reddit - Wed, 04/30/2014 - 10:58pm

Hi I have a query regarding the solveRPN code in The book LYAH , specifically the last line of the complete code which can be found here ( http://lpaste.net/103456), I do understand if read x:[1,2,3] would result in the code being trying to read x as an Int , but why should 'read x:[]' be read as an Int

submitted by nerorevenge
[link] [2 comments]
Categories: Incoming News

Can GHC do purity-to-mutability rewriting?

Haskell on Reddit - Wed, 04/30/2014 - 2:36pm

Let's say you have some record r and you do r { myfield = newvalue }.

With immutability semantic a new value r2 will be created with myfield being set to the newvalue and all other fields being copied over from r; if not shared, r will then get garbage collected at the next occasion (usually very quickly in the next minor GC).

Does there exist some kind of optimisation that recognizes that if r isn't shared (used anywhere else), we could not do the whole allocating and copying and modify myfield in-place?

If yes, how is this optimisation called?

How much of this can GHC already do?

If it cannot do this already, how difficult do you estimate such an analysis to be and what kind of architecture needs to be in place? My first guess is that some kind of liveness analysis should be sufficient.

Note that I'm not talking about the more specific forms of strictness analysis where, say, a tuple (Int, Int) is completely eliminated; I mean precisely the general idea of updating records in-place when there is no other consumer.

submitted by nh2_
[link] [22 comments]
Categories: Incoming News

Douglas M. Auclair (geophf): 'Z' is for the ζ-calculus

Planet Haskell - Wed, 04/30/2014 - 11:32am

'Z' is for ζ-calculus  (pronounced 'zeta-calculus')
So, 'Z' is the last letter of the alphabet, unless you're Greek, then ζ is the sixth letter. The last letter of the Greek alphabet is Ω.
But we already did Ωm-Ωm-Ωm. Ram-Ram-Sita-Ram-Sita-Ram-Ram, so we're good there.
As α denotes the beginning of things, Ω (which on a Mac, you get by selecting option-Z. Fitting, that) denotes the end of things. That's why everybody back then got it when Jesus said: "I am the α and the Ω."
He was the package deal.
But look at you all, doing your A-to-Z challenge faithfully. Did you survive? Are you shell-shocked?
To put this into perspective, my mom was a journalist for local news for our local newspaper. She did the A-to-Z challenge, every day (except Sundays), every month, year-after-year.
So you've completed your A-to-Z challenge. Congratulations! Now, lather, rinse, repeat, every month, for the rest of your life, right?
And you thought one month was hard?
Okay, last entry for my A-to-Z guide book for surveyists, explorers and tourists of mathematics.
Okay, so the lambda calculus is the commingling of the κ-calculus (the contextual calculus) and the ζ-calculus (the control calculus), which we talk about in this post (disclaimer/citation: I get most of my information on the ζ-calculus from Hasegawa's seminal 1995 paper on the κζ-calculi). A good deal of attention, when it has been given at all, has been focused principally on the κ-calculus...
'And why not!' says most. In the κ-calculus you get things done. There's no application, but through composition of (first-order) functions and first-order tuple-types you have numbers, counting, addition, and all the rest come out of that.
With the control calculus you make functions that take functions as arguments.
τ : 1 | ((—) ↠ (—))
So reduction (to your answer) is of the form of the continuation-passing form, for a function in the ζ-calculus f : a → b is of the ζ-term form of:
ζ f : a → (c ↠ b)
Which is to say, the zeta-term, f, take a functionally-typed argument, a, and returns the continuation (c ↠ b).
Now, we got the κ-calculus to work by lifting function arguments to tupled-typed terms, and we saw that with the implementation of addition (here). Now, instead of lifting unit types to tuples, in the ζ-calculus we pass a function on x to the zeta-term by using functional composition, i.e.:
pass(c) o (ζ x . f) ~> f[c/x]
To be able to construct a function in the ζ-calculus we have the function-constructor, code:
given f : c → d and x is the 'constifying function' : 1 → cwe have f o x : 1 → d
from that we have code:
code(f) ≡ ζ x . (f o x) : 1 → (c ↠ d)
Boom! code, then, is a function that creates a function from nothing (the '1' type).
But, okay, what can you do with the ζ-calculus?
Well, since we don't have unit cartesian types, like we have in the κ-calculus, then the answer to that is, well, ... nothing, really.
I mean, it's possible to have function represent unit types then start composing them to build an algebra, but this approach is rather unwieldy. The ζ-calculus exists to show that the λ-calculus is perfectly decomposable into the κ-calculus and the ζ-calculus, and so control is usually mapped out in the ζ-calculus (although some directed flow is possible in the κ-calculus, itself alone, as we've seen) (link).
For example, to create the numbers zero and one are possible in the ζ-calculus, but when we try to represent two (that is, functionally, double application), we get this:
x : 1 → cz : 1 → (c ↠ c)  pass(x) : (c ↠ c) → c  pass(x) o z : 1 → c  -- so we've got our constant continuation (step 1) pass(pass(x) o z) : c ↠ (c → c) -- now we've got a continuation-creator pass(pass(x) o z) o z) : 1 → c ζx (pass (pass(x) o z) o z) : 1 → (c ↠ c) -- continuation-creator for the right-hand sidedupl ≡ ζz . ζx (pass (pass(x) o z) o z))) : 1 → ((c ↠ c) ↠ (c ↠ c))
And there we have it, the dupl(icate) function, or the number two, in the ζ-calculus.
Ugh!
Well, you did say you wanted to see it. I did, too. It's a morbid fascination of mine, watching this train wreck transpire.
The Takeaway here
But that's a reveal. Math isn't necessarily easy (shocker!) nor is it necessarily neat. Here we see in the ζ-calculus that it's difficult to express things (like, for example, the number two, or any unit types).
So, if you've got a correspondence (a provable one), all you have to do is to change categories to one where you can express what you need to simply and cleanly.
The application works here, and now, too.
I'm doing things slow and stupid, because I'm doing them the way I've always done them, the way I see how they're done.
Change categories. Somebody is doing what I'm doing, but doing it well. Somebody isn't doing what I'm doing, because they've moved onto better things — more sublime problems — ... I can do the same thing. All I have to do is change my perspective, how I see things, and then, seeing them anew, I can do what I need to do, what I want to do, simply and cleanly, then translate back down to the way I always do things, ...
Or hey, just move on, after having moved up.
It's called a 'lifting function.' It lifts an object from one category to another one, and in being lifted, the object is changed, and what it can do (its morphisms) are changed. And you can change it back 'down' (the co-ajoint function) or you can stay in the newly-lifted category, or you can lift again.
Math is neat that way, once you see that as a possibility.
Life can be like that, too.
Life is neat that way, once you see that as a possibility.
'Z' is for the ζ-calculus.
And with 'Z' we're all done doing our A-to-Z petit survey of mathematics and logic. I hope you had fun, for I had a blast writing these posts.
Thank you for reading them.
Categories: Offsite Blogs

Hylomorphisms and Treesort

Haskell on Reddit - Wed, 04/30/2014 - 9:30am
Categories: Incoming News