Networking

Data Parallel Bellman-Ford

Submitted by mrd on Thu, 11/29/2007 - 7:48pm.

Graph representation is as an edge-list.


bmf :: Int -> UArr (Int :*: Int :*: Double) -> Int -> UArr Double
bmf n es src = iterate rnd dm0 !! (n - 1)
where
dm0 = toU [ if i == src then 0 else inf | i <- [0 .. n - 1] ]
rnd dm =
updateU dm
(mapU (\ e ->
if distOf dm (destin e) > distOf dm (source e) + weight e
then (destin e) :*: (distOf dm (source e) + weight e)
else (destin e) :*: (distOf dm (destin e)))
es)


source = fstS . fstS
destin = sndS . fstS
weight = sndS
distOf dm u = dm !: u


inf :: Double
inf = 1 / 0

The above code uses NDP but only the sequential portions. In order to get parallelism I need to invoke the Distributed operations. However, there are also Unlifted.Parallel operations which hide the usage of the distributed ops.


bmf :: Int -> UArr (Int :*: Int :*: Double) -> Int -> UArr Double
bmf n es src = iterate (rnd es) dm0 !! (n - 1)
where
dm0 = toU [ if i == src then 0 else inf | i <- [0 .. n - 1] ]


{-# INLINE rnd #-}
rnd :: UArr (Int :*: Int :*: Double) -> UArr Double -> UArr Double
rnd es dm = updateU dm
. mapUP sndS
. filterUP fstS
. mapUP (\ e ->
let d = distOf dm (destin e)
d' = distOf dm (source e) + weight e
in if d > d'
then True :*: (destin e :*: d')
else False :*: (0 :*: 0))
$ es

mapUP is the unlifted parallelized version of mapU. However there's no updateUP. Looking into the code I spotted out a commented out version of updateUP. There are problems when figuring out what to do about multiple concurrent writes to one location. NESL specifies "arbitrary" resolution of conflicting concurrent writes. Unfortunately the commented code has bit-rotted and I haven't successfully managed to fix it.

I also added some filtering to prevent it from getting stuck writing Infinity over a real distance constantly, due to "arbitrary" resolution of conflicting writes in updateU. The iterative function is now factored out into its own toplevel definition, for clarity.

A simple TCP server

Submitted by mrd on Fri, 01/19/2007 - 10:19pm.

A simple TCP server which accepts multiple clients and echos input text back to all those connected. It uses threads to manage multiple handles, and Software Transactional Memory to pass messages.

This text is literate Haskell, and has been tested with ghc 6.6 on Linux/x86. Type annotations are included for didactic purposes.

> module Main where > import Prelude hiding (catch)

Module Network is the simple networking library, presenting a Handle-based interface.

> import Network (listenOn, accept, sClose, Socket, > withSocketsDo, PortID(..)) > import System.IO > import System.Environment (getArgs) > import Control.Exception (finally, catch) > import Control.Concurrent > import Control.Concurrent.STM > import Control.Monad (forM, filterM, liftM, when)

A simple main to parse a port number from the command line, and fire up the server socket.

> main = withSocketsDo $ do > [portStr] <- getArgs > let port = fromIntegral (read portStr :: Int) > servSock <- listenOn $ PortNumber port > putStrLn $ "listening on: " ++ show port > start servSock `finally` sClose servSock
> start servSock = do > acceptChan <- atomically newTChan > forkIO $ acceptLoop servSock acceptChan > mainLoop servSock acceptChan []

acceptLoop manages the server socket, accepting connections, starting client threads, and forwarding the relevant information about them over the channel so the main loop can multiplex it all together.

> type Client = (TChan String, Handle) > > acceptLoop :: Socket -> TChan Client -> IO () > acceptLoop servSock chan = do > (cHandle, host, port) <- accept servSock > cChan <- atomically newTChan > cTID <- forkIO $ clientLoop cHandle cChan > atomically $ writeTChan chan (cChan, cHandle) > acceptLoop servSock chan

As before, each client gets a loop which reads from the handle and pumps the data right into a channel. However, this time, exception handling is done per-thread; if a client disconnects we just want the thread to die silently. A more clever implementation might have a more structured channel which allows it to indicate when the client disconnects.

> clientLoop :: Handle -> TChan String -> IO () > clientLoop handle chan = > listenLoop (hGetLine handle) chan > `catch` (const $ return ()) > `finally` hClose handle > listenLoop :: IO a -> TChan a -> IO () > listenLoop act chan = > sequence_ (repeat (act >>= atomically . writeTChan chan))

STM conveniently allows composition of actions which makes custom tailoring of library code a snap. Here, I've added an additional action to check the status of the acceptChan along with all the clients. The acceptChan has a different type than any of the client channels, so I separate it from the others using an Either data type for simplicity. `fmap` here acts very much like (.), the functional composition operator.

> mainLoop :: Socket -> TChan Client -> [Client] -> IO () > mainLoop servSock acceptChan clients = do > r <- atomically $ (Left `fmap` readTChan acceptChan) > `orElse` > (Right `fmap` tselect clients) > case r of > Left (ch,h) -> do > putStrLn "new client" > mainLoop servSock acceptChan $ (ch,h):clients > Right (line,_) -> do > putStrLn $ "data: " ++ line

In addition to sending the data out to every client, this loop catches any errors from writing to handles and excludes that client from the list.

> clients' <- forM clients $ > \(ch,h) -> do > hPutStrLn h line > hFlush h > return [(ch,h)] > `catch` const (hClose h >> return []) > let dropped = length $ filter null clients' > when (dropped > 0) $ > putStrLn ("clients lost: " ++ show dropped) > mainLoop servSock acceptChan $ concat clients'

tselect is a function which multiplexes any number of TChans. It will return the data from whichever TChan it can read, along with the "key" value that can be supplied in the pair. This takes advantage of the STM combinators orElse and retry by applying them to a list of actions constructed around the TChans.

> tselect :: [(TChan a, t)] -> STM (a, t) > tselect = foldl orElse retry > . map (\(ch, ty) -> (flip (,) ty) `fmap` readTChan ch)

This code demonstrates a basic TCP server as well as a more generally applicable function tselect. It serves as an example of the strength and simplicity of the Software Transactional Memory model, and of network IO in Haskell.

A simple TCP client

Submitted by mrd on Thu, 01/18/2007 - 8:02pm.

A simple example network client showing how to multiplex the reading of lines from the remote peer and the local user, using Software Transactional Memory to do message-passing and light-weight threads.

This text is literate Haskell, and has been tested with ghc 6.6 on Linux/x86. Type annotations are included for didactic purposes.


> module Main where > import Prelude hiding (catch)

Module Network is the simple networking library, presenting a Handle-based interface.

> import Network (connectTo, withSocketsDo, PortID(..)) > import System.IO > import System.IO.Error (isEOFError) > import System.Environment (getArgs) > import Control.Exception (finally, catch, Exception(..)) > import Control.Concurrent > import Control.Concurrent.STM

main parses host and port from the command line and connects to it. Then it calls the start function with the socket handle. An error handler is defined which prints out exceptions, except for EOF. Finally, the socket is ensured to be closed.

withSocketsDo is required on Windows platforms, but does no harm otherwise.

> main = withSocketsDo $ do > [host, portStr] <- getArgs

PortNumbers are an instance of Num, but not Read. So, we read it as an Int, and then generalize to class Num using fromIntegral.

> let port = fromIntegral (read portStr :: Int) > sock <- connectTo host $ PortNumber port > start sock `catch` handler `finally` hClose sock > where > handler (IOException e) > | isEOFError e = return () > handler e = putStrLn $ show e

start takes care of the creation of threads and channels to communicate between them. Each thread spawned is responsible for listening to a given handle, and forwarding any communications received along the channel. Notice, in particular, how this listening task has been abstracted into a higher-order monadic function. The main thread is then used as the central "coordinating" loop, as discussed below.

> start :: Handle -> IO () > start sock = do > netChan <- atomically newTChan > userChan <- atomically newTChan > netTID <- spawn $ listenLoop (hGetLine sock) netChan > userTID <- spawn $ listenLoop getLine userChan > mainLoop sock netChan userChan

spawn is a small wrapper around forkIO which adds a small thread-specific exception handler that simply passes any exceptions along to the main thread.

(Note: myThreadId is GHC-specific)

> spawn :: IO () -> IO ThreadId > spawn act = do > mainTID <- myThreadId > forkIO $ act `catch` throwTo mainTID

listenLoop pipes the output of calling an action repeatedly into a channel.

Read literally: listenLoop repeats forever, in sequence, the action of invoking act and then atomically writing its value to the channel.

> listenLoop :: IO a -> TChan a -> IO () > listenLoop act chan = > sequence_ (repeat (act >>= atomically . writeTChan chan))

Here is an explicit-recursion version of the above:

listenLoop = do v <- action atomically $ writeTChan chan v listenLoop action chan

Understanding why both versions of listenLoop are equivalent will help you understand that monadic actions in Haskell are first-class values.

mainLoop demonstrates the usage of a simple select function which awaits input from one of two channels.

> mainLoop :: Handle -> TChan String -> TChan String -> IO () > mainLoop sock netChan userChan = do > input <- atomically $ select netChan userChan > case input of > -- from netChan, to user > Left str -> putStrLn str >> hFlush stdout > -- from userChan, to net > Right str -> hPutStrLn sock str >> hFlush sock > mainLoop sock netChan userChan

select multiplexes two TChans using the STM combinator orElse.

Read plainly, it states that ch1 should be consulted first, or else, ch2 should be read. Any values from ch1 are tagged with Left and any values from ch2 are tagged with Right.

The reason why this works seamlessly is because STM will keep track of which channels it has attempted to read. If both have nothing available right now, then STM knows it can block until one of the two channels has data ready.

> select :: TChan a -> TChan b -> STM (Either a b) > select ch1 ch2 = do > a <- readTChan ch1; return (Left a) > `orElse` do > b <- readTChan ch2; return (Right b)

Transcript:


$ ghc --make Client1.lhs [1 of 1] Compiling Main ( Client1.lhs, Client1.o ) Linking Client1 ... $ ./Client1 localhost 25 220 localhost ESMTP Exim 3.36 helo localhost 250 localhost Hello mrd at localhost [127.0.0.1] quit 221 localhost closing connection $

This code has demonstrated a simple TCP client which can receive and transmit lines in an interactive session. The implementation used light-weight threads to multiplex handles and Software Transactional Memory for inter-thread communication. Several re-usable functions were defined which show the expressive power and simplicity of STM and higher-order monadic functions.

getContents with gnutls...

Submitted by jmuk on Sun, 08/20/2006 - 8:04pm.

GetContents like actions are hard to implement with gnutls. hsgnutls prepares `tlsRecv', but it blocks when there are nothing to be read. tlsCheckPending is said to check the lenth of `pending' (readable) buffer in gnutls, but it always returns 0.

I read `gnutls-cli', a telnet like gnutls command because it can read until there are something to be read. Then, I found that gnutls-cli uses select(2) to know whether reading is ready or not. Oops...

Next, I keep the Handle for tlsClient in TlsSession data structure, and use hReady :: Handle -> IO Bool to check if the tls session is readable. It seems to succeed when I test in ghci, but fails when using in other actions (like bsGetContents). The reason will be that the state of hReady cannot change so rapidly. For example,
*TLSStream> bsPutCrLf s (BS.pack "a001 CAPABILITY") >> hReady h >>= print
False
*TLSStream> hReady h >>= print
True

Then, I try to use hWaitForInput :: Handle -> Int -> IO Bool because it can wait some period. Now, it succeed to check with the waiting time of 500 miliseconds.

Are there any other (elegant) implementation of `GetContent' with gnutls?

packrat parsing, and idea of `ParserT'

Submitted by jmuk on Wed, 08/16/2006 - 1:05am.

I use Packrat Parsing for the parser of HaskellNet.
see, http://darcs.haskell.org/SoC/haskellnet/Text/
and, more detailed information of Packrat Parsing are seen at http://pdos.csail.mit.edu/~baford/packrat/

Packrat Parsing is easy to use and develop.

Of course, there already exists Parsec. However, Parsec cannot be applied with ByteString. Parsec are intended for lists of tokens. With Packrat Parsing, we must define dvChar -- a function to calculate the `next' character of a stream -- by ourselves, so it is easier to use with ByteString. It's great.

Then, I want a kind of `ParserT'. For example, IMAP server response may includes `status updates' informations. Server may respond as follows:

* 22 EXPUNGE
* 23 EXISTS
* 3 RECENT
* 14 FETCH (FLAGS (\Seen \Deleted))
* CAPABILITY IMAP4rev1 STARTTLS AUTH=GSSAPI LOGINDISABLED
abcd OK CAPABILITY completed

This response is primarily for `capability' of the server, but this response also includes the `new' information of current mailbox.

Currently, I split these responses as (ServerResponse, MailboxUpdate, ResponseData). ServerResponse is OK, BAD, NO and so on. ResponseData is [String] in this case. MailboxUpdate is Recent 3 and so on. After parsed, the connection data updates its mailbox information.

If we have MonadTrans of Parser, we can update the mailbox information at the time of parsing `3 Recent'. Then, there are no need to prepare MailboxUpdate type and such confusing structure.
In other cases, `ParserT' seems to be useful.

I thought this problem a little but it seems difficult. Are there any ideas?
I'd like to think later...

What is the best abstraction for network socket?

Submitted by jmuk on Thu, 07/27/2006 - 5:32am.

There are many stream like data structure proposed now. At first, the standard library already contains Network and Network.Socket. Network module expresses socket as a Handle and Network.Socket module does it as `Socket' type, which is not compatible with Handle.

Then, Streams and (http://haskell.org/haskellwiki/Library/Streams) and SSC are proposed (http://yogimo.sakura.ne.jp/ssc/). Which is more proper library?
Streams has no consideration about networking and SSC has. But, Socket of SSC is just an instance of BlockPort (using Ptr a), which is not a good abstraction for sockets, IMHO.
Is SSC better choise?

Then, I'd like to handle sockets with ByteString for performance reason. And I will have to deal with SSL/TLS using hsgnutls or such like libraries.

Now, HaskellNet is written by Network.Stream and Network.TCP originally came from HTTP. They are good abstraction about socket, but are not the best one when considering about ByteString-ization and SSL/TLS.

Please let me know about other implementation or consideration about this topic if you know.

type definitions of IMAP

Submitted by jmuk on Sun, 06/11/2006 - 10:06am.

In naive implementation, IMAP requires same constructor names in different context. For example, RECENT is used at the SEARCH command, STATUS command, and in STATUS response. In this case, I would like to define as follows...

  type Mailbox = String
  data IMAPCommand = STATUS Mailbox [StatusQuery]
                   | SEARCH [SearchQuery]
                         :
  data StatusQuery = RECENT
                   | MESSAGES
                   |  :
  data SearchQuery = RECENT
                   | SEEN
                   |  :
  data MailboxData = RECENT Integer
                   | SEEN Integer
                   |  :

In addition to this, IMAP commands are separated as some groups. For example, UID command is used for SEARCH, FETCH, STORE, and COPY. How can I describe this relation in the type definition?
By using polymorphic variant of OCaml, these can be described easily and elegantly.

In conclusion, such relations may be described using not only type definitions, but also function definitions.

IMAP specs and typing

Submitted by jmuk on Thu, 06/08/2006 - 2:13am.

IMAP, Internet Message Access Protocol, is a fairly complecated protocol (comparing with POP3). Its specification can be read as RFC 3501.

I worry about the typing of IMAP commands. In normal protocol, the type of a client command can be written such as `Connection -> Request -> IO Response'. However, IMAP server may also return a status update data.

The server program can notify its clients with a status update untagged data. Assume that there is a client connecting to a server and the server may receive a new mail during the connection, for example. In normal, such newly received mail cannot be read from the client because the connection is established before the reception of the mail. However, the server may notify the client as that `it has a new mail'. Any commands may have such status update data.

Therefore, the type of a IMAP command should be `Connection -> Request -> IO (Response, Maybe StatusUpate)', but such typing will be annoying in many case...
Certainly, such typing can be hidden with StateT or WriterT. But is it clever approach?

BTW, I test the behavior of IMAP with Courier-IMAP, an implementation of IMAP, but it will not send any status update data with a command except NOOP. mmm....

Netcat Tutorial

Submitted by Adam Palmer on Sat, 06/11/2005 - 2:20pm.

Netcat Tutorial
A LearnSecurityOnline.com Article By Adam Palmer

http://www.learnsecurityonline.com © Learn Security Online, Inc. 2004-2005

Contents

Introduction 3
Netcat Syntax 4
Netcat Installation 6
Simple File Transfer 7
Tar 9
Simple socket reply 10
inetd 11
talking to syslogd -r 12
Internetworking Basics 13
nc-e 14
Scanning 15
Spoofing 16
Simple response service 17
Advanced Uses 18
Windows Command Shell 19
Unauthorized Proxying 20
Cryptcat 20
Final Thoughts 21
Command cheat sheet 22

Introduction

What is Netcat?

"Netcat is a simple Unix utility which reads and writes data across network connections, using TCP or UDP protocol. It is designed to be a reliable "back-end" tool that can be used directly or easily driven by other programs and scripts. At the same time, it is a feature-rich network debugging and exploration tool, since it can create almost any kind of connection you would need and has several interesting built-in capabilities. Netcat, or "nc" as the actual program is named, should have been supplied long ago as
another one of those cryptic but standard Unix tools." Taken from the README of the netcat source tree, this description sums up the uses of netcat perfectly.

Netcats homepage is: http://netcat.sourceforge.net

Throughout this tutorial, I will be giving examples on Linux systems. The official Netcat homepage makes no reference to Windows systems, however I have successfully built Netcat from source under Cygwin, and you can find a Win32 copy built by @Stake from:

http://www.atstake.com/research/tools/network_utilities/nc11nt.zip and all examples used
below are fully supported under Windows.

http://www.learnsecurityonline.com © Learn Security Online, Inc. 2004-2005

Netcat Syntax

adam@adamp:~$nc -h

[v1.10]
connect to

somewhere: nc [-options] hostname

port[s] [ports] ..
.
listen for
inbound: nc -l -p port [-options]
[hostname] [port]

options:
-e prog programto exec after connect [dangerous!!]
-b allow

broadcasts
-g gateway source-routing hop point[s], up to 8
-G num source-routing

pointer: 4, 8, 12, ...
-h this
cruft
-i secs delayinterval for lines sent, ports scanned
-l listen
mode, for inbound connects
-n numeric-onlyIP addresses, no DNS
-o file hex
dump of traffic
-p port localport number
-r randomize
local and remote ports
-q secs quitafter EOF on stdin and delay of secs
-s addr local
source address
-t answer
TELNET negotiation
-u UDP
mode

-v verbose
[use twice to be more verbose]

-w secs timeout
for connects and final net reads

-z zero-I/
O
mode [used for scanning]

port
numbers can be individual or ranges: lo-hi [inclusive]

http://www.learnsecurityonline.com © Learn Security Online, Inc. 2004-2005

Netcat Installation

I will cover here three installation methods.

a) On a debian or similar machine:

apt-get install netcat will do the trick:

adamp:~# apt-get install netcatReading Package Lists... DoneBuilding Dependency Tree... Done

The following NEW packages will be installed:

netcat
0 packages upgraded, 1 newly installed, 0 toremove and 0 not upgraded.
Need to get 63.3kB of archives. After unpacking190kB will be used.
Get:1 http://http.us.debian.org stable/mainnetcat 1.10-21 [63.3kB]
Fetched 63.3kB in 2s (27.9kB/s)
Selecting previously deselected package netcat.
(Readingdatabase ... 39433 files and directories currently installed.)
Unpacking netcat (from.../netcat_1.10-21_i386.deb) ...
Setting up netcat (1.10-21) ...
adamp:~#

b) And for those that prefer RPMs:

rpm Uvh netcat-version.rpm

c) And for those that prefer the source:

We will start by wgeting the source:

wget http://osdn.dl.sourceforge.net/sourceforge/netcat/netcat-0.7.1.tar.gz

We will now untar, cd to the directory we have untarred the source codeto, and run the configure script.

adam@adamp:~$ tar -xzf netcat-0.7.1.tar.gzadam@adamp:~$ cd netcat-0.7.1adam@adamp:~/netcat-0.7.1$ ./configure

The configure script should run through with no trouble, as netcat has very few dependencies.
We then run make:

adam@adamp:~/netcat-0.7.1$ make

This will run through and will compile your source, which again should complete simply and successfully. You
can then run make install if you have the necessary privileges, or you could simply run src/netcat which will have
been built after a successful make. At this point, you should now have a successful build of netcat
somewhere on your system.

http://www.learnsecurityonline.com © Learn Security Online, Inc. 2004-2005

Simple File Transfer

So as an example, I will start two copies of netcat on the same machine locally:

adam@adamp:~$ netcat -l -p 1111

Here, using the -l switch, we are able to specify that netcat should go into listen mode i.e. to listen on
the specified port. Using p 1111 we are able to specify that we are using port 1111. To summarize,
netcat will sit and listen for TCP connections on port 1111 and print any data it receives out to the
screen. In another window we start netcat as:

adam@adamp:~$ netcat 127.0.0.1 1111

This will connect to host 127.0.0.1 (Locally) on port 1111.
We are now able to have a full two way data transmission, in Window 1:

adam@adamp:~$ netcat -l -p 1111This message was typed in WINDOW1This message was typed in WINDOW2Now I'm going to end communication with ^C (Ctrl-C)
adam@adamp:~$

And in Window 2:

adam@adamp:~$ netcat 127.0.0.1 1111This message was typed in WINDOW1This message was typed in WINDOW2Now I'm going to end communication with ^C (Ctrl-C)
adam@adamp:~$

This is the most basic use of netcat described. Here, we are using a BASH shell, and thus we may pipe |
data to and from netcat, as well as using the redirection (>, >>, <, <<) to allow netcat to integrate into
the shell environment. We will now examine using netcat with one of the redirection operators. Lets
say we wanted to simply transmit a plaintext file. In one window, we will start netcat as:

adam@adamp:~$ netcat -l -p 1111 > outputfile

This will run netcat with the same parameters specified above, except it will redirect
all text received into outputfile.

adam@adamp:~$ echo > infile << EOF
> This is a test file.
> I am going to attempt to transmit this.
> Using Netcat.
> EOF
adam@adamp:~
$

Here, we have created some text in a file, and this is the file we are going to attempt to transmit:

adam@adamp:~$ cat infile | netcat 127.0.0.1 1111 q 10adam@adamp:~$

http://www.learnsecurityonline.com © Learn Security Online, Inc. 2004-2005

Hopefully this has now been transmitted to the otherside:
adam@adamp:~$ cat outputfile
This is a test file.

I am going to attempt to transmit this.
Using Netcat.
adam@adamp:~
$

And here we can confirm that it has. The -q 10 in the command line will quit after EOF (Otherwise
netcat will hang waiting for more input for cat and we will have to terminate it manually). The
parameter 10 causes it to quit after 10 seconds anyway.

Tar

Now, there is no reason why we can’t integrate tar and netcat together, and use this to transmit a
directory across a netcat socket:

On one side: tar zcfp - /path/to/directory | nc -w 3 127.0.0.1 1234

The tar statement before the pipe tars and compresses (using gzip) every file within that directory,
before printing its output to stdout (The screen). It is then caught by the pipe, and piped to nc which in
this example, connects to 127.0.0.1 on port 1234 and sends it the data which would normally hit the
screen. The w 3 switch causes nc to allow for a 3 second timeout (In the event of a temporary
disconnection or similar).

On the other side: nc -l -p 1234 | tar xvfpz

This will listen on port 1234 for a connection, and will pass any data received to tar. Using the option v
we can print out filenames to screen:

Simple Socket Reply

With what we have learned so far, we are easily able to get netcat to listen in on a socket, and pump out
any data we wish when it receives a connection.

As an example:

while true; do echo "Leave me alone" | netcat -l -p 1234 w10; done

Consider this line. Firstly lets examine echo "Leave me alone" | netcat -l -p 1234 -w10

What we are doing here, is listening in on port 1234 with a wait time of 10 seconds. If/when we receive
a connection, pipe the results of echo "Leave me alone" to netcat. The w 10 is necessary, as otherwise
any connection made in will remain open forever. We can also optionally add a v in to the netcat
command line which will give us verbose information, i.e. who is connecting.

Every time a connection times out (either with the w 10 command line switch, or because a connection
has been made and then closed), netcat will exit. As this is not what we want, we put the command line
within a standard BASH: while CONDITION; do STATEMENT; done clause, which when the
condition is set to true will run forever.

Inetd

If you build netcat with GAPING_SECURITY_HOLE defined, you can use it as an "inetd" substitute
to test experimental network servers that would otherwise run under "inetd".

A script or program will have its input and output hooked to the network the same way, perhaps sans
some fancier signal handling.

Given that most network services do not bind to a particular local address, whether they are under
"inetd" or not, it is possible for netcat avoid the "address already in use" error by binding to a specific
address.

This lets you [as root, for low ports] place netcat "in the way" of a standard service, since inbound
connections are generally sent to such specifically-bound listeners first and fall back to the ones bound
to "any".

This allows for a one-off experimental simulation of some service, without having to screw around
with inetd.conf. Running with -v turned on and collecting a connection log from standard error is
recommended.

Netcat as well can make an outbound connection and then run a program or script on the originating
end, with input and output connected to the same network port.

This "inverse inetd" capability could enhance the backup-server concept described above or help
facilitate things such as a "network dialback" concept.

The possibilities are many and varied here; if such things are intended as security mechanisms, it may
be best to modify netcat specifically for the purpose instead of wrapping such functions in scripts.

Speaking of inetd, netcat will function perfectly well *under* inetd as a TCP connection redirector for
inbound services, like a "plug-gw" without the authentication step.

This is very useful for doing stuff like redirecting traffic through your firewall out to other places like
web servers and mail hubs, while posing no risk to the firewall machine itself.

Put netcat behind inetd and tcp_wrappers, perhaps thusly:

www stream tcp nowait nobody /etc/tcpd /bin/nc -w 3 realwww 80

and you have a simple and effective "application relay" with access control and logging. Note use of
the wait time as a "safety" in case realwww isn't reachable or the calling user aborts the connection --
otherwise the relay may hang there forever.

Inetd/tcp_wrappers and netcat information, courtesy of: http://www.spyder-fonix.com/netcat.html

Talking to syslogd -r

Syslog Daemons running with the r switch log not only their own hosts data but accept remote UDP
broadcasts. They listen in on UDP port 514.

"echo '<0>message' | nc -w 1 -u loggerhost 514"

If loggerhost is running syslogd r and can accept your messages.

Note the -u switch here, to put netcat into UDP mode. Specifying the <0> before your message ensures
that your message receives top priority within syslog (kern.emerg)

Internetworking Basics

For the purposes of this section, machine refers to an x86 compatible PC with a connection to the
Internet through some means, terminated by a standardized TCP/IP stack.

Each machine on the Internet today comes shipped with a standard, compatible TCP/IP stack. This
stack guarantees the use of 65535 ports, and IPv4 protocol compatibility.

Below we can see the OSI model. This explains in terms of 7 layers, how data is constructed at one
host and received at the next.

In short; Data is constructed on the left by an application, encodes it with a transport (TCP) which takes
it over the network (IP), resolves MACs of local devices (Data Link) and then passes a constructed
packet to the network card which transmits (Physical) it over the wire (at which point the opposite
happens at the other end).

You may have intelligent devices such as switches along the way. These for example may be wise up
to layer 5 for example and not only route according to MAC address (Layer 2) but inspect and firewall
packets based on findings up to Layer 5 (Simple firewalling) or even Layer 7 (Packet inspection).

"The OSI, or Open System Interconnection, model defines a networking framework for implementing
protocols in seven layers. Control is passed from one layer to the next, starting at the application layer
in one station, proceeding to the bottom layer, over the channel to the next station and back up the
hierarchy." (Courtesy of: http://webopedia.internet.com/quick_ref/OSI_Layers.asp)

nc -e

We have already discussed the basics of redirection with netcat. Netcat has a e switch which we can
use to execute a program on connection. There are a couple of viable and legitimate uses for this, i.e.
running as nc e v called by the inetd wrapper, which we can use to view traffic and information on
users connecting to wrapped daemons, however the most common use which we will explore here is
using it to redirect to and from /bin/bash or similar shell, for both good and bad.

One method could be this:

adam@adamp:~$ nc -v -e '/bin/bash' -l -p 1234 –tlistening on [any] 1234 ...
connect to [127.0.0.1] from localhost [127.0.0.1] 51210

In one window, and a simple telnet localhost 1234 in another window:

adam@adamp:~$ telnet 127.0.0.1 1234
Trying 127.0.0.1..
.
Connected to 127.0.0.1.
Escape character is '^]'
.
echo Test
Test
^
]
telnet>

Scanning

The scanning features of netcat can be used against yours or your friends networks to get useful
information about which hosts have certain ports open. You can also send a precompiled data file to
each. For example:

Echo EXIT | nc -w 1 127.0.0.1 20-250 500-600 5990-7000

Will scan 127.0.0.1 on ports 20-250, 500-600 and 5990-7000. Every port that it finds is open, it will
pipe the output of echo "EXIT" being the word "EXIT" to that port.
The results are as follows:

(For the sanity of my server, I have blocked out a number of parts from certain service banners.)
And now with UDP scanning: nc -v -w 1 127.0.0.1 u 20-250 500-600 5990-7000 we receive:

adam@adamp:~$ nc -u -v -w 1 127.0.0.1 20-250 500-600 5990-7000localhost [127.0.0.1] 250 (?) openadam@adamp:~$

-v was to put netcat into verbose mode, and u was telling netcat to fall into UDP mode.

Spoofing

"Your TCP spoofing possibilities are mostly limited to destinations you can source-route to, while
locally bound to your phony address.

Many sites block source-routed packets these days for precisely this reason.

If your kernel does oddball things when sending source-routed packets, try moving the pointer
around with -G. You may also have to fiddle with the routing on your own
machine before you start receiving packets back.

Warning: some machines still send out traffic using the source address of the outbound interface,
regardless of your binding, especially in the case of localhost.

Check first. If you can open a connection but then get no data back from it, the target host is probably
killing the IP options on its end [this is an option inside TCP wrappers and several other packages],
which happens after the 3-way handshake is completed.

If you send some data and observe the "send-q" side of "netstat" for that connection increasing but
never getting sent, that's another symptom. Beware: if Sendmail 8.7.x detects a source-routed SMTP
connection, it extracts the hop list and sticks it in the Received: header!"

http://www.spyder-fonix.com/netcat.html

Spoofing is a useful technique, as is source routing.

Source routing is almost obsolete now, and the majority of routers filter out source routed packets.
Source routing in a nutshell is basically setting the route that the packet will take at the source, and
storing that information along with the packet.

Normally, each router makes its own mind up as to where a packet will get routed, and follows its
predefined routing tables. If we have access to all routers between our device and the target device
(which can be one machine if youre talking about your local LAN server), then we are able to modify
the routing entries on those devices, bind a phoney address to our machine and source route packets to
the intended destination.

Spoofing is where we modify the source address of a packet so that the recipient believes it came from
a different address. There are two problems with this;


A number of clever ISP routers will drop packets with incorrect source addresses.

If the destination host does get to receive your spoofed packet, it will send data back to the
spoofed address (instead of ours). This does have a number of uses however in the example of
ICMP ping flooding a host and spoofing the source address to Microsoft.com (as a theoretical
example).

Simple Response Service

echo -e "GET http://www.google.com HTTP/1.0\n\n" | nc w 5www.google.com 80

We make a connection to google.com on port 80 (Web server port), and put in an HTTP request for
http://www.google.com. At this point, we are presented with the HTML spurted out by the web server.
We can pipe this to "| less" or similar or even our favourite HTML interpreter.

Take a look at this example, and you will see what we have done here. In one instance we have created
an HTML file webfrontend and we now pipe that HTML to any incoming connection to netcat on port
1111. We then make a connection on the larger window, using lynx http://127.0.0.1:1111 and we have
made ourselves a tiny http server, possibly could be used as a holding page server or something similar.

Advanced Uses

Now we'll set up a server netcat to listen on port 1111. We'll also set up a client netcat to talk to the real
web server on port 81. By getting them to pass all data they receive to each other, together they form a
proxy; something that sits in the middle of a network connection. Here are the commands we use:

mknod backpipe p

nc -l -p 1111 0backpipe

Because bash pipes only carry data in one direction, we need to provide a way to carry the responses as
well. We can create a pipe on the local filesystem to carry the data in the backwards direction with the
mknod command; this only needs to be run once.

Requests coming into the proxy from the client arrive at the first nc, listening on port 1111. They get
handed off to the "tee" command, which logs them to the inflow file, then continue on to the second nc
command which hands them off to the real web server. When a response comes back from the server, it
arrives back at the second nc command, gets logged in the second tee command to the outflow file, and
then gets pushed into the backpipe pipe on the local filesystem. Since the first netcat is listening to that
pipe, these responses get handed to that first netcat, which then dutifully gives them back to the original
client.

While the above example is for watching tcp streams going to and from a web server, the above
technique is useful for watching any tcp connection. In fact, since nc also works with udp packets
something telnet can't do - it should be possible to even set up udp proxies this way.

Windows Command Shell

As we can see from the image above, we have started netcat with options of l p 1234 e
"c:\windows\system32\cmd.exe". These are the same options as with the Unix shell, and this should
theoretically start a cmd.exe shell listening in on port 1234:

As you see from above, this has succeeded. Netcat and program execution for Windows can be used in
exactly the same way.

Unauthorised Proxying

Assume you are an administrator of a Linux router. Using the methods above, as well as your iptables
software, you can proxy a users outgoing connection through your nc proxy. Using iptables with the j
DNAT target and the j REDIRECT target, you can transparently proxy outgoing connections through
to any other ports you want, and what better to use than your nc proxy?

Cryptcat

Cryptcat can be found at: http://sourceforge.net/projects/cryptcat/ and is the ultimate companion for
Netcat. It includes a lightweight version of Netcat, featuring encrypted transport properties. (Just for
those superbly paranoid!). Useful for encrypting communications out of a network.

Final Thoughts

If I was given one tool on a freshly installed PC, I would ask for Netcat. Due to its versatility and its
huge range of uses, it can be used as a transfer tool, a scanning tool, a server, a proxy and so much
more. I have put down everything useful I can think of, and welcome any further suggestions directed
to adam@apnicsolutions.com

Command Cheat Sheet

The following are the most useful uses of netcat:

For windows nc d can be used to detach from the console.

nc -l -p [port] will create a simple listening tcp port. Add u to put into UDP mode.
nc -e [program] To redirect stdin/stdout from program.
nc -w [timeout] To set a timeout before netcat automatically quits. (Used within a loop usually)
program | nc To pipe output of program to netcat
nc | program To pipe output of netcat to program
nc -h Help sheet
nc -v To put into verbose mode, or use v v to put into ultra-verbose mode!
nc -g or nc G Source routing flags
nc -t Use telnet negotiation (If performing telnet negotiation)
.
nc -o [file] Hex dump traffic to file
nc -z No I/O (Used for scanning ports)