Recently in Programming Category

This is a tale of two approaches to regular expression matching. One of them is in widespread use in the standard interpreters for many languages, including Perl. The other is used only in a few places, notably most implementations of awk and grep. The two approaches have wildly different performance characteristics: .... (Hier geht's zum eigentlichen Artikel)

grep3p.png
grep4p.png

Der Inhalt des Artikels ist relativ interessant. Zwecks Dokumentation blogge ich ihn hiermit.

Es muss halt nicht immer Perl sein. Es geht auch mal mit AWK.

C++ Reference

| No Comments | No TrackBacks

Ein Klassiker der Referenzen der Programmiersprache C++ ist die Website www.cppreference.com. Die Site ist immer Empfehlenswert (gerade da sie relativ übersichtlich gestaltet ist und man sich relativ schnell zurecht findet). Fragen betreffend C++ im IRC kann ich meist (seit mehreren Jahren) mit einem Verweis darauf beantworten. :)

Standard ML und Haskell #2

| No Comments | No TrackBacks
Lindig2.jpegNochmal ein kleiner Vergleich zwischen Standard ML und Haskell. Dieses Mal geht es um Funktionen höherer Ordnung. Die Syntax ist fast die Selbe. Jedoch Haskell ist ein kleines bischen weniger verbose. Hier sind 4 Funktionen, jeweils in einer Standard ML Version (oben) und in einer Haskell Version (unten):

fun make_map_fn f1 = fn (x,y) => f1 x :: y
make_map_fn f1 = \x y -> f1 x : y

fun make_filter_fn f1 = 
     fn (x,y) => if f1 x then x :: y else y
make_filter_fn f1 = 
     \x y -> if f1 then x : y else y

fun my_map f l = foldr (make_map_fn f) [] l
my_map f l = foldr (make_map_fn f) [] l

fun my_filter f l = foldr (make_filter_fn f) [] l
my_filter f l = foldr (make_filter_fn f) [] l
Die Haskell Syntax ist darauf optimiert so wenig Boilerplate Code wie möglich zu produzieren. Standard ML ist da allerdings auch nicht gerade schlecht (die Funktionen my_map und my_filter haben in Haskell und in SML quasi die selbe Syntax. SML benutzt lediglich noch das Schlüsselwort fun).

Die interaktive SML shell lässt zu Wünschen übrig. So versuch ich vergebens einen Befehl die aktuell geladene Datei erneut einzulesen. In Haskell (ghci) geht das z.B. mit dem Befehl :r (r für reload) wie folgt:

*Main> :r
[1 of 1] Compiling Main             ( temp.hs, interpreted )
Ok, modules loaded: Main.

Bei SML hingegen muss ich mit Strg+D den interaktiven Prompt verlassen und die Bash-Histoy benutzen um "sml meinesourcecodedatei.sml" erneut einzulesen. Zwar gibt es einen sml-Befehl eine Sourcecodedatei einzulesen, aber der ist zu lang um ihn jedes Mal erneut eingeben zu können (Mein sml hat auch keine History-Funktion).

Standard ML und Haskell

| No Comments | No TrackBacks
Derzeit erarbeite ich mir die Grundlagen der funktionalen Programmiersprache Standard ML (Studium - Fortgeschrittene Konzepte funktionaler Programmierung). Da ich schon ein bischen Haskell kann, konnte ich es mir nicht nehmen die Aufgaben auch in einer Haskell-Version zu implementieren. Wie man sieht sind sich SML und Haskell (zumindest bisher in den Grundlagen) sehr ähnlich. Allerdings ist die Syntax von Haskell an einigen Stellen etwas "fortgeschrittener". Es werden in Haskell insgesamt etwas weniger Schlüsselwörter (z.B. val, end, fun, fn...) und Kommas in der Syntax verwendet. Ausserdem gibt es in Haskell die (optionale) Möglichkeit die Typen einer Funktion explizit hinzuschreiben. Was leserlicher ist und zudem Programmierfehler vermeidet. Was ich bisher bei SML vermisse sind auch die sog. Pattern Guards die es in Haskell gibt. I.A. gefällt mir bisher Haskell besser als SML, u.A. auch da Haskell sich "pur funktional" schimpft während SML auch von imperativen Konzepte explizit Gebrauch macht. Aber soweit bin ich in SML noch nicht. Hier ein paar Funktionen in SML und in Haskell im Vergleich:

Standard ML:

datatype ''a multi 
  = EMPTY
  | ELEM of ''a
  | UNION of ''a multi * ''a multi

Haskell:

data (Eq a) => Multi a
  = Empty
  | Elem a
  | Union (Multi a) (Multi a) 
  deriving Show

Standard ML:

fun number (EMPTY) _ = 0
  | number (ELEM x) w = if x = w then 1 else 0
  | number (UNION (x,y)) w = (number x w) + (number y w)

fun test_number w = 
  number (UNION (
    EMPTY, 
    UNION (
      ELEM 4,
      UNION (
        ELEM 6,
        UNION (
          UNION (
            ELEM 4,
            ELEM 4
          ),
        EMPTY)
      )
    )
  )) w

Haskell:

number Empty _ = 0
number (Elem x) w = if x == w then 1 else 0

test_number w = 
  number (Union 
    Empty
    (Union 
      (Elem 4)
      (Union
        (Elem 6)
        (Union
          (Union
            (Elem 4)
            (Elem 4)
          )
        Empty)
      )
    )
  ) w

Standard ML:

fun simplify (UNION (x,y)) = 
    let fun is_empty (EMPTY) = true
        | is_empty _ = false
       val x' = simplify x
       val y' = simplify y
     in if (is_empty x') andalso (is_empty y')
        then EMPTY
       else if (is_empty x')
        then y'
       else if (is_empty y')
        then x'
       else UNION (x', y')
    end
  | simplify x = x

Haskell:

simplify (Union x y) 
    | (isEmpty x') && (isEmpty y') = Empty
    | isEmpty x' = y'
    | isEmpty y' = x'
    | otherwise = Union x' y'
  where 
    isEmpty Empty = True
    isEmpty _ = False
    x' = simplify x
    y' = simplify y
simplify x = x

Standard ML:

fun delete_all m w =
  let fun delete_all' (ELEM x) = if x = w then EMPTY else ELEM x
      | delete_all' (UNION (x,y)) = UNION (delete_all' x, delete_all' y)
      | delete_all' x = x
   in simplify (delete_all' m)
  end

Haskell:

delete_all m w = simplify (delete_all' m)
  where
    delete_all' (Elem x) = if x == w then Empty else Elem x
    delete_all' (Union x y) = Union (delete_all' x) (delete_all' y)
    delete_all' x = x


Standard ML:

fun delete_one m w =
  let fun delete_one' (UNION (x,y)) = 
      let val (x', deleted) = delete_one' x 
       in if deleted 
        then (UNION (x', y), deleted)
        else let val (y', deleted) = delete_one' y 
           in (UNION (x, y'), deleted)
          end
        end
      | delete_one' (ELEM x) = 
        if x = w then (EMPTY, true) else (ELEM x, false)
      | delete_one' x = (x, false)
      val (m', _) = delete_one' m 
    in simplify m'
  end

Haskell:

delete_one m w = do
    let (m', _) = delete_one' m 
      simplify m'
  where
    delete_one' (Union x y) = 
      let (x', deleted) = delete_one' x 
       in if deleted 
        then (Union x' y, deleted)
        else let (y', deleted) = delete_one' y 
             in (Union x y', deleted)
    delete_one' (Elem x) = 
        if x == w then (Empty, True) else (Elem x, False)
    delete_one' x = (x, False)

Monadic Perl

| No Comments | No TrackBacks
Nach C++ hab ich auch etwas gefunden, wo Monaden in Perl implementiert wurden. Es funktioniert, aber in Haskell wirkt die Implementierung von Monaden noch etwas eleganter. Aber das Resultat kommt dem in Haskell recht nahe:

sub ev
{
    my $term = $_[0];
    if($term =~ /^$con/)
    {
        return Return($1);
    }
    elsif($term=~/^$div/)
    {
        my $numer=$1; my $denom=$3;
        return DO {
                    my $n <- ev($numer);
                    my $d <- ev($denom);
                    my $g <- tick();
                    Return( $n / $d );
                  }
    }else{die "Syntax Error: $term";}
}

Lambda the Ultimate

| No Comments | No TrackBacks

Hiermit empfehle ich Lambda the Ultimate. Ein Weblog mit dem Thema Programmiersprachen. Viele Beiträge sind etwas abgehoben. Aber Andere wiederum recht interessant. Lohnt sich zeitweise reinzuschnuppern.

I've mentioned this book once before.

wizard.jpgStructure and Interpretation of Computer Programs has been MIT's introductory pre-professional computer science subject since 1981. It emphasizes the role of computer languages as vehicles for expressing knowledge and it presents basic principles of abstraction and modularity, together with essential techniques for designing and implementing computer languages. This course has had a worldwide impact on computer science curricula over the past two decades.

You can find the online videos and the free online book right here. For the case the ftp servers are busy you can download them from my own pub ftp instead (ftp.buetow.org/pub).

'Programming Haskell' online lectures

| 1 Comment | No TrackBacks

haskell.png I found a great series of free haskell lectures online at http://www.cs.nott.ac.uk/~gmh/book.html#videos. The referent is going through all the chapters of the book 'Programming Haskell'. The videos are pretty good and atm I am enjoying watching them. You should give them a try as well! I think you don't even need to buy the book in order to understand everything.

Update: I put those videos on my pub FTP server for download: ftp://ftp.buetow.org/pub/OnlineLectures/ProgrammingHaskell/

Redesign of the Fype Interpreter

| No Comments | No TrackBacks

fype_small.pngAfter reading Structure and Interpretation of Computer Programs I decided to rethink the interpreter of Fype. Fype is a small scripting language invented and implemented (in C) by me. I decided to take the code-base of Fype and to rewrite the core of its interpreter-part in order to use a Scheme/Lisp-like syntax. which is much easier to parse. The syntax is simpler and much more powerful now. So far it is now possible to define functions and sub-functions and to display all involving frames. Here is an example:

(def (test)
   (say "This is test")
   (def (test2) 
      (say "I am in test2" "And test3 will be defined next!")
      (def (test3)
         (say "Displaying all frames now:")
         (show-frames))
      (test3))
   (test2))
(test)

The invocation of the interpreter prints out the following result:

This is test
I am in test2
And test3 will be defined next!
Displaying all frames now:
FRAME(id=3) 0:
FRAME(id=2) 1:
+ST_LAMBDA(name=test3;args=)
( ( say Displaying all frames now: ) ( show-frames ) )
FRAME(id=1) 2:
+ST_LAMBDA(name=test2;args=)
( ( say I am in test2 And test3 will be defined next! ) ( def ( test3 ) ( say Displaying all frames now: ) ( show-frames ) ) ( test3 ) )
FRAME(id=0) 3:
+ST_LAMBDA(name=test;args=)
( ( say This is test ) ( def ( test2 ) ( say I am in test2 And test3 will be defined next! ) ( def ( test3 ) ( say Displaying all frames now: ) ( show-frames ) ) ( test3 ) ) ( test2 ) )

Now what's left to do: Implement a few more built in functions. And we have a new Scheme/Lisp-like language with a very very small footprint available :)

Btw.: This version of fype aims to be pure-functional and with lazy evaluation.

pwgrep v0.4 is out

| No Comments | No TrackBacks

I just released pwgrep v0.4. What's new? Besides of storing passwords into a single file, pwgrep can now also be used for storing a collection of files, which is very usefull for storing certificate files etc. Like passwords, all files stored within pwgrep are encrypted using GPG. The command pwfls will list all files currently stored in your database:

7.png

pwfadd adds a specific file to the database:

8.png
9.png

pwfls will show you that it has been added successfully:

10.png
11.png

File deletion can be accomplished using pwfdel:

12.png

In general you can decrypt/store/encrypt any specific file format (as well as binary files).

Introducing pwgrep.buetow.org

| No Comments | No TrackBacks

After introducing pwgrep itself I am now introducing the homepage of pwgrep. Its address is pwgrep.buetow.org.

It contains all informations about pwgrep :)

A few months ago I was looking for an easy tool for resolving host names and IP adresses. Well, I didn't have time to manage to write such a tool by myself. I was thinking: "Why making it so complicated?". So I came up with the following solution:

#!/bin/sh
exec xterm -geometry 60x10 -title 'resolving hosts or ips' \
     -e bash -c 'while read line; do [ ! -z $line ] \
     && host $line; done'

And I told my window manager (FVWM) all xterms with the title 'resolving hosts or ips' to always stay on top and to be sticky. I think this is an easy tool for resolving hosts and IPs. I just have to use Xorgs copy and paste mechanism (3rd mouse button) to copy those into the window in order to get resolved :)

resolver.png

The easiest solutions are the best solutions (at least in this case)!

Introducing pwgrep (Update)

| No Comments | No TrackBacks

Thumbnail image for seprian-logo-3.pngIn order to manage all my secret passwords I wrote myself a small bash/awk script which manages a database file using encryption of GnuPG. (Source of pwgrep v0.2)

The database file is divided in several records. Each record begins with its name followed by several lines holding all the secret informations. The (actually very simple) format of the database file is as follows:

some record name here
      after a tabulator some secret informations          
      more secret informations
another record name here ..... 
      secret username: foo
      secret password is bla
      you can write as many secret infos as you wish
.
.
secret stuff
      password: hello world
      username: mr. universe

The database is not stored in plain text. It is encrypted using GnuPG (database.gpg).

I can only search for the record names of a database file. For example if I want to see my secret username and password which is stored in the database.gpg file it will look like this:

1.png

After entering the password of my secret GnuPG key I will receive the informations requested:

2.png

pwgrep will print out automatically all records matching my search string. Not only the first one it finds.

I can use pwedit for the case I want to add something to the database or just to edit/delete something of the current database:

3.png

4.png

After editing, pwgrep will automatically wipe all temporally files securely and it will commit the new version into the versioning system (In this case subversion is being used. But others can be configured as well). pwgrep is using Vim (with swapping and backuping disabled) in order to edit the database file. If you want to use a different editor, you should make sure NOT to produce temporally files. If you produce temporally files, at least they should get wiped securely from the hard disk.

5.png

6b.png

If you want to look up your secret ebay stuff, just search for it with

pwgrep ebay

If found, pwgrep uses destroy or shred for wiping files. If none of those commands are found, pwgrep checks if the current operating system is FreeBSD. In this case, rm -P can be used as well. Otherwise an error will occur.

Update: Uploaded v0.2 of pwgrep. Before editing the newest version will be checked out before. This will avoid versioning conflicts. The tool also creates local snapshot files of the encypted database (for the case you lost internet connection and you want to roll back to the previous version) And a few other enhancements have been built in.

CXG Next Generation online

| No Comments | No TrackBacks

cxg.gifMy brother, Florian Bütow, has released the first version of OpenCXG which is a new pastebin script written from scratch in PHP. OpenCXG is the successor of the older CXG pastebin which was based on mod_python. The latter however was broken after a few major operating system and library upgrades.

OpenCXG still does not have all features which the older CXG used to have. However the plans for the future can be read on the OpenCXG homepage.

Xerl Revision 182

| No Comments | No TrackBacks

xerl.buetow.org.png Xerl is an open source website template engine (TE) with some features of a Content Management System (CMS) programmed in object oriented Perl and using XML.

It now supports the redirect:* directive. Which allows me to configure HTTP redirects from one host to another without using mod_rewrite or .htaccess. redirect:* is working like the alias:* directive of the previous feature upgrade.

One small step for Xerl - one giant step against duplicated content.

Xerl Revision 170

| No Comments | No TrackBacks

xerl.buetow.org.pngI've committed Xerl Revision number 170 to the subversion repository. Xerl is an open source website template engine (TE) with some features of a Content Management System (CMS) programmed in object oriented Perl and using XML.

Enhancements made:

  • Host aliases can now be configured using an alias:hostname file. E.g.:
    echo www.buetow.org > hosts/alias:buetow.org
    would create the host buetow.org as an alias to www.buetow.org. Means: buetow.org shows the same content as www.buetow.org. My hosts/ directory now looks like this (alias:* are host alias definitions and the rest are directories for the "real" virtual hosts):
    alias:buetow.org               fype.buetow.org
    alias:butow.org                irssi.buetow.org
    alias:dslvpnrouter.buetow.org  jsmstrade.buetow.org
    alias:pb-labs.com              lan.buetow.org
    alias:perl9.org                netcalendar.buetow.org
    alias:www.butow.org            niduterm.buetow.org
    alias:www.pb-labs.com          paul.buetow.org
    alias:www.xn--btow-0ra.com     perlchat.buetow.org
    alias:www.xn--btow-0ra.org     stud.buetow.org
    alias:xn--btow-0ra.com         use.perl9.org
    alias:xn--btow-0ra.org         vpndslrouter.buetow.org
    awksite.buetow.org             vs-sim.buetow.org
    bg.buetow.org                  www.buetow.org
    blogs.buetow.org               www.perl9.org
    calculator.buetow.org          xerl.buetow.org
    curses.buetow.org              ychat.buetow.org
    default                        yhttpd.buetow.org
    fuzzybot.buetow.org
    Because of duplicated content, the next version of Xerl may use a HTTP redirect instead (search engines). The old way of defining aliases still works as well:
    echo buetow.org=www.buetow.org >> conf.txt
    However this method is not the preferred anymore. conf.txt needs to get restructured anyway.
  • Initial support for plugins
  • Some code review including more documentations about the coding style to use.

Xerl is work in progress! It has no real documentation and no release available atm. This is mostly a hobby project fitting my own needs in having dynamic websites.

I am looking for a way of doing fuzzy string matching using the Haskell programming language. I want to parse the user's text input (which is a normal natural english sentence). The user may will have errors in his text, which should get recognized and replaced automatically by my program. I think the easiest way would be calculating the 'Levenshtein distance' to all known words and use the word with the smallest distance. Although, this is not really fuzzy logic (fuzzy logic depends, in my view, more on the context of the sentence which is written), I think it is a good starting point in order to get an initial version working. According to the Wikibook the Haskell implementation would be as easy as this:

levenshtein :: String- > String -> Int
levenshtein s t = 
    d!!(length s)!!(length t) 
    where d = [[distance m n|n<-[0..length t]]|m<-[0..length s]]
          distance i 0 = i
          distance 0 j = j
          distance i j = minimum [d!!(i-1)!!j+1, 
                                 d!!i!!(j-1)+1, d!!(i-1)!!(j-1) 
                                  + (if s!!(i-1)==t!!(j-1) 
                                     then 0 else 1)]

This method may be too slow if I've to compare each word with about serveral thousands words. Hrrm.

Or using another implementation, because:
The implementations of the Levenshtein algorithm on this page are illustrative only. Applications will, in most cases, use implementations which use heap allocations sparingly, in particular when large lists of words are compared to each other.

Yet another way would be implementing a Fuzzy Logic Library (as learned in the Neuronal Fuzzy Networks lecture @ Aachen University of Applied Sciences) from scratch.

Progress in Haskell

| 1 Comment | No TrackBacks

My first real program written in Haskell is growing. It is forming valid english sentences.

makeSentences = do
	let pronoun = SgI 
        in print $ (pronounForm pronoun) 
           +++ (futureForm pronoun $ IrregularVerb 
                         "go" "went" "gone") 
           +++ (superlative $ RegularAdjective "fast")
	let pronoun = SgHe 
        in print $ (pronounForm pronoun) 
           +++ (presentForm pronoun $ IrregularVerb 
                         "go" "went" "gone") 
           +++ (comparative $ RegularAdjective "fast")
	let pronoun = PlWe 
        in print $ (pronounForm pronoun) 
           +++ (pastForm pronoun $ RegularVerb "watch") 
           +++ (comparative $ IrregularAdjective 
                         "good" "better" "best")

The output of this small program is as follows:

*Main> makeSentences
"I will go the fastest"
"he goes faster"
"we watched better"

real-world-haskell.jpgSo far the program handles pronouns, regular verbs, irregular verbs, regular adjectives and irregular adjectives with all of their exceptions. The program is less then 200L. Since I am used to C/C++/Perl and Java, programming in Haskell forces me to rethink the way of programming. It's a great experience. I still 've much to learn. And my Real World Haskell book I've only finished half yet :)

Learning functional programming

| 4 Comments | 1 TrackBack

The last weeks I started (slowly but constantly) to learn functional programming using Haskell. First, I started with the functional programming practices (available as videos) of the RWTH Aachen (University). However those practices aren't so good if you don't have the lecture itself as well. Afterwards I was reading most of the Yet Another Haskell Tutorial and did most of the practices listed inside. I did this until I found out about the book Real World Haskell, which is also available for free as HTML pages. I decided a hard cover book is much nicer and it is supporting the authors, so I ordered myself a copy of this. While waiting for it, I found a 15 year old book about Informatics in my shelf.

vorlesung-informatik.jpg

The last ~150 pages teach how to functional program using the Gofer Programming Language. While waiting for Real World Haskell I am reading how to program in Gofer at the moment. Gofer can be seen as a very light version of Haskell. So if I learn Gofer, then I also learn Haskell. Today, Haskell is 100% compatible to Gofer plus more.

One of my long term goals may be reimplementing my AI IRC Bot in Haskell. At the moment, the bot is written in Perl.

JSMSTrade v0.2 released

| No Comments | 1 TrackBack

I just released the second (v0.2 to be more specific) version of JSMSTrade. JSMSTrade is a very small and handy client (programmed in Java) for SMSTRADE.de for sending SMS messages.

jsmstrade.buetow.org.png

The changes are as follows: The application is now using the english language and is not in german anymore. Also a small bugfix has been included: The input area will not freeze after typing more than 160 chars.

Enjoy ;)

Update: v0.3 has been released now too. This version only includes a major bugfix. Sadly, v0.2 was not able to deliver any SMS messages. This has been fixed by now :)

About this Archive

This page is an archive of recent entries in the Programming category.

Podcasts is the previous category.

Python is the next category.

Find recent content on the main index or look in the archives to find all content.