Clan x86

Technical (Development, Security, etc.) => General Programming => Botdev => Topic started by: nslay on September 08, 2012, 05:02:26 pm

Title: Creating BNX clone for IRC
Post by: nslay on September 08, 2012, 05:02:26 pm
I'm going to attempt to recreate as much of BNX as possible for IRC.

Let me know if you're interested. I've created an SVN repository for it and committed some reference materials (namely BNX 1.03 files and Greetbot files).

I think I'll first create a rudimentary battle.net CHAT emulator (designed for one-on-one CHAT). This way we can use Wine to try BNX out. Though BNX is actually quite well documented.

I think the best choice of implementation language is C++ and we can use libevent (http://libevent.org) for portable event-driven timers and socket multiplexing.

Again, if you're interested, just PM me a SVN username/password and get a dynamic domain name for your development machine (so I can add you to the firewall).

RFC 1459 (http://www.faqs.org/rfcs/rfc1459.html)
Title: Re: Creating BNX clone for IRC
Post by: nslay on September 09, 2012, 05:00:00 pm
I've decided to make a base IrcClient class to take care of all the basic IRC details.
So far, I've implemented all the socket code, the connect/reconnect code, basic event dispatching, and line buffering. Everything is non-blocking and revolves around libevent (this allows several instances of IrcClient to coexist).

Next is to implement more of the get/set functions (only basic ones exist), IRC protocol parsing and high level IRC event dispatching.

One of the most irritating aspects of IRC is the begin, content, and end numerics. It makes the programming a little trickier since you have to keep track of the object of interest currently being constructed. For example, the channel list is sent in multiple messages. You can be in several channels at a time as well as querying channel lists of which you are not in.

In the distant future, I'll create a derived BnxBot class that just implements the BNX behavior.
Title: Re: Creating BNX clone for IRC
Post by: nslay on September 10, 2012, 11:38:40 pm
I wrote the IRC protocol parsing today ... I tried to stick to the RFC as close as possible. However, I'm not very familiar with the IRC protocol

I'm actually surprised it could be written so concisely! Here's what I wrote:
Code: [Select]
char * PopToken(char *&str) {
        str += strspn(str, " ");

        char *tmp = str;

        str += strcspn(str, " ");
        if (*str != '\0') {
                *str++ = '\0';
                str += strspn(str, " ");
        }

        return tmp;
}

void IrcClient::OnProcessLine(char *line) {
        const char *pPrefix = NULL, *pCommand = NULL, *pParams[16] = { NULL };
        unsigned int numParams = 0;

        puts(line);

        if (*line == ':')
                pPrefix = PopToken(line)+1;

        pCommand = PopToken(line);

        while (*line != '\0' && numParams < 16) {
                if (*line == ':') {
                        pParams[numParams++] = line+1;
                        break;
                }   

                pParams[numParams++] = PopToken(line);
        }   

        if (isdigit(pCommand[0])) {
                // This is probably a numeric

                int numeric = strtol(pCommand,NULL,10);

                OnNumeric(numeric, pParams, numParams);

                return;
        }   

        if (!strcmp(pCommand,"PING")) {
                OnPing(pPrefix, pParams[0]);
        }   

}

Does it look OK? Do you suppose there is any case that I don't handle adequately?
Title: Re: Creating BNX clone for IRC
Post by: nslay on September 14, 2012, 02:03:15 am
Well, I setup all the callbacks for all word commands and I setup automatic nickname handling (collision and in-use sorts of issues).

I also found RFC 2812 (http://tools.ietf.org/html/rfc2812) and added more #defines for newer RPL and ERR numerics.

I'm thinking about how to implement the BNX wildcards. Here's a snippit from BNX.TXT
Quote
? * - matches 0 or more characters of any type
? ? - matches exactly one character of any type
? % - matches any number of non-space characters
? ~ - matches at least one space character
? \ - the literal escape character
UNIX-like systems usually provide fnmatch(3) (http://www.freebsd.org/cgi/man.cgi?query=fnmatch) which only covers * and ?.

This sort of wildcard matching will also need to be implemented for the access system anyway (for hostmasks).

I'm also thinking about easy and portable configuration file formats. INI seems to be very simple and sufficiently flexible for this application. There is even a library called iniparser (http://ndevilla.free.fr/iniparser/).

Any comments?
Title: Re: Creating BNX clone for IRC
Post by: rabbit on September 16, 2012, 11:12:17 am
Well, I setup all the callbacks for all word commands and I setup automatic nickname handling (collision and in-use sorts of issues).

I also found RFC 2812 (http://tools.ietf.org/html/rfc2812) and added more #defines for newer RPL and ERR numerics.

I'm thinking about how to implement the BNX wildcards. Here's a snippit from BNX.TXT
Quote
? * - matches 0 or more characters of any type
? ? - matches exactly one character of any type
? % - matches any number of non-space characters
? ~ - matches at least one space character
? \ - the literal escape character
UNIX-like systems usually provide fnmatch(3) (http://www.freebsd.org/cgi/man.cgi?query=fnmatch) which only covers * and ?.

This sort of wildcard matching will also need to be implemented for the access system anyway (for hostmasks).

I'm also thinking about easy and portable configuration file formats. INI seems to be very simple and sufficiently flexible for this application. There is even a library called iniparser (http://ndevilla.free.fr/iniparser/).

Any comments?
Convert them to regular expressions.
Title: Re: Creating BNX clone for IRC
Post by: nslay on September 18, 2012, 12:29:26 am
With regex in C++11, I guess it wouldn't be an issue of portability.

The usual DOS wildcards still need to be supported for hostmasks (nobody writes hostmasks in regex).
Title: Re: Creating BNX clone for IRC
Post by: nslay on October 06, 2012, 06:57:29 pm
It's been a while. I've been very busy these past two weeks.
Anyway, I spent some time today to implement the IRC wildcards.

Here's what I wrote:
Code: [Select]
bool IrcMatch(const char *pPattern, const char *pString) {
        if (pPattern == NULL || pString == NULL)
                return false;

        const char *pPatternLast = NULL, *pStringLast = NULL;

        do {
                if (*pPattern == '*') {
                        pPattern += strspn(pPattern,"*");
                        pPatternLast = pPattern;
                        pStringLast = pString;
                }   
     

                if (*pPattern == '?') {
                        if (*pString == '\0') {
                                if (pPatternLast == NULL || *pStringLast == '\0')
                                        return false;

                                pPattern = pPatternLast;
                                pString = ++pStringLast;
                                continue;
                        }   
                }   
                else {
                        if (*pPattern == '\\')
                                ++pPattern;

                        if (*pPattern != *pString) {
                                if (pPatternLast == NULL || *pStringLast == '\0')
                                        return false;

                                pPattern = pPatternLast;
                                pString = ++pStringLast;
                                continue;
                        }   
                }   

                if (*pPattern != '\0')
                        ++pPattern;

                if (*pString != '\0')
                        ++pString;

        } while (*pString != '\0');

        pPattern += strspn(pPattern,"*");

        return *pPattern == '\0' && *pString == '\0';
}

So as an explanation:
This is pretty straightforward. However, when you add the BNX wildcards % and ~, then you have to do something more sophisticated. You need to keep something like a stack of pattern partitions to match. Unlike IRC wildcards, you may have to revisit an earlier partition since the % and ~ require some specific structure.
Title: Re: Creating BNX clone for IRC
Post by: nslay on October 07, 2012, 05:03:50 pm
I wrote the response engine today. It's quite elegant. Have a look:

Code: [Select]
class BnxResponseEngine {
public:
        BnxResponseEngine() {
                AddDefaultResponse("Please go on.");
                AddDefaultResponse("You must be joking!");
                AddDefaultResponse("Are you talkin' to me?");
                AddDefaultResponse("Get outta town!");
        }   

        bool LoadFromStream(std::istream &is);
        void SaveToStream(std::ostream &os) const;

        void AddDefaultResponse(const std::string &strMessage) {
                m_vDefaultResponses.push_back(strMessage);
        }   

        const std::string & ComputeResponse(const std::string &strMessage) const {
                std::vector<BnxResponseRule>::const_iterator itr;

                itr = std::find(m_vRules.begin(), m_vRules.end(), strMessage);

                if (itr == m_vRules.end())
                        return m_vDefaultResponses[rand() % m_vDefaultResponses.size()];

                return itr->ComputeResponse();
        }   

        void Reset() {
                m_vRules.clear();
        }   

private:
        std::vector<BnxResponseRule> m_vRules;
        std::vector<std::string> m_vDefaultResponses;
};

By the way, the BNX executable contains no obvious trace of those default responses (at least from strings(1) point of view). The author must have done something clever to prevent hex editing.

Here are some screenshots of some classic responses:

As you can see, "/me" is not properly handled. Actually, there is no "/me" command on IRC. Only the kludge called CTCP. So, I'll have to add some battle.net command to CTCP conversion ...

EDIT: Use thumbnails instead.

(http://imageshack.us/a/img803/5099/bnxbot2.th.png) (http://imageshack.us/photo/my-images/803/bnxbot2.png/)
(http://imageshack.us/a/img585/9023/bnxbot.th.png) (http://imageshack.us/photo/my-images/585/bnxbot.png/)

Title: Re: Creating BNX clone for IRC
Post by: nslay on October 13, 2012, 05:19:40 pm
I wrote a CtcpEncoder, CtcpDecoder and added some limited support to BnxBot. It now responds to the CTCP VERSION command (though, it doesn't have any real versioning yet).

Today, I created a sourceforge project for it:
http://sourceforge.net/projects/ircbnx/

While sourceforge doesn't think there are any files, you can check them out with subversion (I recommend TortoiseSVN on Windows).

For Windows:
EDIT: Redundant
Some porting is still necessary despite using libevent2. For one, FreeBSD's libstdc++ doesn't support C++11 std::regex and so I used POSIX regex. BnxResponseRule is the only place where regex is used and it would probably be trivial to port this to C++11 std::regex (e.g. I think VS 2010 and 2012 have this).


A list of things that remain to be done:

After I get through these, I'll mark the project as Alpha.
Title: Re: Creating BNX clone for IRC
Post by: nslay on October 14, 2012, 02:36:05 am
Getting close to finishing a baseline.

I implemented the following
Title: Re: Creating BNX clone for IRC
Post by: nslay on October 14, 2012, 05:04:16 pm
I apparently also need a send queue (which is not obvious). I guess it doesn't surprise me.

See here:
http://forums.unrealircd.com/viewtopic.php?f=9&t=7654

I also prepared a test sequence for the original BNX to see which commands could be accessed at which levels. Here's the script:
Code: [Select]
#!/bin/sh

MakeCommands() {
        echo "2010 NAME nslay"
        echo "1007 CHANNEL \"cheese\""
        echo "1001 USER nslay 0012 [CHAT]"
        echo "1002 JOIN test 0010 [CHAT]"

        #set -x
        wine bnx.exe &

        sleep 5

        for i in `seq 0 20`
        do
                level=`expr 100 - ${i} '*' 5`
                user="enslay${i}"

                echo "*** level = ${level}" 1>&2

                echo "*** login" 1>&2
                echo "1004 WHISPER ${user} 0010 \"login passwd\""
                sleep 3

                echo "*** say" 1>&2
                echo "1004 WHISPER ${user} 0010 \"say test\""
                sleep 3

                echo "*** kick" 1>&2
                echo "1004 WHISPER ${user} 0010 \"kick test\""
                sleep 6

                echo "*** ban" 1>&2
                echo "1004 WHISPER ${user} 0010 \"ban test\""
                sleep 6

                echo "*** unban" 1>&2
                echo "1004 WHISPER ${user} 0010 \"unban test\""
                sleep 6

                echo "*** shitadd" 1>&2
                echo "1004 WHISPER ${user} 0010 \"shitadd test\""
                sleep 3

                echo "*** shitdel" 1>&2
                echo "1004 WHISPER ${user} 0010 \"shitdel test\""
                sleep 3

                echo "*** shitlist" 1>&2
                echo "1004 WHISPER ${user} 0010 \"shitlist\""
                sleep 6

                echo "*** join" 1>&2
                echo "1004 WHISPER ${user} 0010 \"join cheese2\""
                sleep 6

                echo "*** join" 1>&2
                echo "1004 WHISPER ${user} 0010 \"join cheese\""
                sleep 6

                echo "*** shutup" 1>&2
                echo "1004 WHISPER ${user} 0010 \"shutup\""
                sleep 3

                echo "*** chatter" 1>&2
                echo "1004 WHISPER ${user} 0010 \"chatter\""
                sleep 3

                echo "*** squelch" 1>&2
                echo "1004 WHISPER ${user} 0010 \"squelch test\""
                sleep 6

                echo "*** unsquelch" 1>&2
                echo "1004 WHISPER ${user} 0010 \"unsquelch test\""
                sleep 6

               echo "*** splatterkick" 1>&2
                echo "1004 WHISPER ${user} 0010 \"splatterkick test\""
                sleep 15

                echo "*** userlist" 1>&2
                echo "1004 WHISPER ${user} 0010 \"userlist test\""
                sleep 90

                echo "*** voteban" 1>&2
                echo "1004 WHISPER ${user} 0010 \"voteban test\""
                sleep 60
        done

        echo "1004 WHISPER enslay 0010 \"login passwd\""
        echo "1004 WHISPER enslay 0010 \"shutdown\""

        wait
}

MakeCommands | nc -l 6112


Nifty what you can do with netcat!
Title: Re: Creating BNX clone for IRC
Post by: nslay on October 26, 2012, 01:46:26 am
I'd say it's about 85% done. Maybe in at most a couple weeks it will be alpha-quality. I'll port it to Windows, Linux and OS X (if I can find access to one).

You can see the current status here (https://sourceforge.net/p/ircbnx/wiki/Home/#implementation-status).

The first release is meant to be an accurate recreation of the original BNX. Maybe after that, I can add some extra features.
Title: Re: Creating BNX clone for IRC
Post by: nslay on November 02, 2012, 12:31:36 am
Good news, IRCBNX is practically done. I implemented ban, unban and splatterkick today (among many commands in the past week). That leaves one remaining command, voteban as well as some BNX-specific behavior.

If you have the time and interest, do have a look over the source (Look under "Browser SVN" on sourceforge) and let me know if there are any design improvements that could be made.
Title: Re: Creating BNX clone for IRC
Post by: nslay on November 04, 2012, 01:43:39 am
I added voteban support and configuration file support. I ended up having to write my own INI parser since many existing libraries consider # the beginning of a comment. I also ported it to Windows which actually went surprisingly smoothly. Very little had to be modified and most of that little bit was isolated to the IrcClient base class.

I have a single bot (it's configurable for several bots ... not that it really matters) sitting in #ircbnx on irc.freenode.net for testing purposes.

It's not terribly exciting ... but there you have it.

The project is now promoted to alpha status though it still has some remaining missing details and hopefully I can get around to making some Linux, Windows and OS X binaries.
Title: Re: Creating BNX clone for IRC
Post by: nslay on November 08, 2012, 09:55:26 am
Check this out:
http://forum.valhallalegends.com/index.php/topic,18164.msg185900.html#msg185900

Invert demystified the origins of BNX.
Title: Re: Creating BNX clone for IRC
Post by: nslay on November 10, 2012, 10:23:49 pm
So, I'm ready to make some binaries, though I'm not sure whether I should statically link the libraries or not on Windows. Opinions?

A static ircbnx binary for 64 bit Windows is ~7MB ... kind of clunky. Still, that incorporates PCRE, libevent2 and libstdc++. Though, I like the idea of simply using ircbnx without needing to find runtimes ... and I'm sure any users (if any) would appreciate that too.

Also, I hear Visual Studio's license does not allow open source development ... that you must protect the source code when you distribute the executable. Any truth to this?

Either way, I've been able to produce mingw32/64 compiled binaries. I might try using clang + libcxx instead of mingw and GNU libstdc++ for fun.

Comments?
Title: Re: Creating BNX clone for IRC
Post by: Newby on November 29, 2012, 03:04:45 pm
The lack of dependency fetching is a nice draw to the software, statically linking is cool.  And I'm not sure about the VS license and open source confliction...  (I figure I'd respond since you seem actively into this and nobody else is remarking on it.. I've been reading it without responding for the most part.  It's a really cool project, I just wish I had free time to appreciate it. :()
Title: Re: Creating BNX clone for IRC
Post by: nslay on November 29, 2012, 06:18:35 pm
The lack of dependency fetching is a nice draw to the software, statically linking is cool.  And I'm not sure about the VS license and open source confliction...  (I figure I'd respond since you seem actively into this and nobody else is remarking on it.. I've been reading it without responding for the most part.  It's a really cool project, I just wish I had free time to appreciate it. :()
The bot is practically finished. Every major feature of BNX (except designate) is implemented (with some extra IRC-specific functionality).

For the past 2 weeks, one instance (nickname is enslay) has been sitting on three networks: binaryninjas (#dojo), betawarz (#dk187,#beta), freenode (#ircbnx).

No problems yet.

Though, I'd like to add server-side ban list management (since IRC severely restricts the number of ban entries) to wrap up the first version.

In future versions, I think I need to embed a scripting engine into it to make it competitive with other bots. It also needs more secure access (e.g. over DCC) since server ops can potentially sniff messages you send through the server (though, the server really ought to be trusted ...).

I'll keep you (the forum) posted.
Title: Re: Creating BNX clone for IRC
Post by: nslay on December 31, 2012, 02:17:14 am
I released 1.0. It's mostly just tidying up what was there and writing documentation. No server-side ban list management yet. It's much trickier than I anticipated.

I compiled 32 bit and 64 bit (why?) versions with MingW. Everything is statically linked, so hopefully it works correctly out of the box. I'm especially curious if it runs on Windows XP. The "latest" download is the source code, so you'll have to navigate to the "ircbnx" folder to get the executable.

Once I remember how to use rpmbuilder, I'll package it in RPM too ...

In all honesty, I wanted the bot to background like a proper daemon but I'm not sure how to deal with this in Windows (I don't really want it to be a service). So, it'll just show an empty Command Prompt like the old one ...

Feedback and suggestions are appreciated ... if you care.


Title: Re: Creating BNX clone for IRC
Post by: rabbit on January 08, 2013, 03:11:07 pm
Run it windowless with an icon in the system tray (or whatever they are calling it nowadays)?
Title: Re: Creating BNX clone for IRC
Post by: nslay on January 12, 2013, 02:40:03 pm
Run it windowless with an icon in the system tray (or whatever they are calling it nowadays)?

Thanks for the suggestion. As I am largely unfamiliar with Windows API, can you refer (if you know) to an MSDN reference?
Title: Re: Creating BNX clone for IRC
Post by: rabbit on January 14, 2013, 03:09:31 pm
http://msdn.microsoft.com/en-us/library/windows/desktop/bb762159(v=vs.85).aspx
Title: Re: Creating BNX clone for IRC
Post by: nslay on January 19, 2013, 01:12:46 pm
http://msdn.microsoft.com/en-us/library/windows/desktop/bb762159(v=vs.85).aspx

Much appreciated!

Random:

Here's a picture of a Flemish Giant (http://en.wikipedia.org/wiki/Flemish_Giant) rabbit:
(http://upload.wikimedia.org/wikipedia/commons/thumb/d/dc/Runt_and_Paxie.jpg/799px-Runt_and_Paxie.jpg)
Title: Re: Creating BNX clone for IRC
Post by: nslay on January 26, 2013, 12:41:56 pm
In addition to the basic BNX features, I've recently added (among a lot of internal code cleanups and added standards compliance)

I plan to add a seen command too. Any features you'd like to see in IRCBNX?

Why who and seen you ask? Firstly, everyone on most IRC networks I've frequented are automatically invisible (mode +i), so you can't actually make any practical use of IRC's WHO command unless you're in the channel you're querying! Secondly, IRC's WHOWAS is apparently sometimes disabled (e.g. on freenode). I originally thought, "Gee, IRC has WHOWAS, why bother with seen?"

IRC folk apparently have trouble understanding how who is useful. Aside of good debugging information, who allows you to list the users in a channel you may not be able to join to ask for help (for a variety of reasons: banned, channel limit, invite-only, keyed, etc...). And well, a historical Battle.net reason for who was that you could determine if a server was split. I remember those days when you could see users in a channel that no one else in the channel could see and bots' who command was useful for checking if the bot was on a partially split server. The same applies to IRC I think ... IRC has serious splitting problems like early Battle.net (I almost wonder if Battle.net 1.0 was a heavily modified IRCd!).

I'm not so certain I'll embed a scripting engine now (many people seem to find it a PLUS that it doesn't embed a scripting engine!). At this point, I'm looking for simplicity ... I look at eggdrop, for example, and it feels like you have to be an expert to set it up. But it's IRC bot ... who the fuck wants to read documentation to set up something like that? No, Battle.net bots were superior in that respect ... very simple, functional, and entertaining. These IRC guys are crazy!

I'll have to pick Invert's brain about his modified BNX's botnet support (if he remembers) ... is a botnet really useful enough to be troubled to implement it?

I'm also conflicted on SSL support. I think Unreal is the only IRCd that supports SSL and with the export/customs laws, I'm not really sure it's worth the trouble. It'd be pretty easy to setup with OpenSSL though. Comments?
Title: Re: Creating BNX clone for IRC
Post by: nslay on May 26, 2013, 01:26:58 pm
I've added a seen and lastseen command just yesterday. Normally, IRC supports WHOWAS, but it apparently can be disabled by the server (e.g. freenode). seen merely tells you if the bot has seen someone, when and in which channel the user was last seen. The seen list is updated on join and privmsg (to a channel). A timer writes the updated seen list to file once every 5 minutes. lastseen tells you who has been seen in a channel in the past day.

I'll try to address the daemonization problem today. It would be nice to be able to daemonize the bot on both Windows (without being a service) and on UNIX-like systems.

I'll try to release 1.2 in early June.
Title: Re: Creating BNX clone for IRC
Post by: nslay on September 08, 2013, 02:35:02 pm
http://msdn.microsoft.com/en-us/library/windows/desktop/bb762159(v=vs.85).aspx
I've been too busy to work on this lately. I did implement Daemonize() for Unix though, just not Windows. I'll try to tackle this today.

Any suggestions for a tray icon?

EDIT:
I'd like to use the CHAT icon from the old days. I'm just concerned about copyrights ... unless you can find this icon in a collection of Blizzard-released website fan kits?
Title: Re: Creating BNX clone for IRC
Post by: nslay on September 08, 2013, 02:40:37 pm
A distant plan for this project is a learning-based NLP response system. It's been my life long dream to create a bot that can talk with some amount of intelligence ... of course, this wouldn't really be BNX-like anymore (BNX-like IRCBNX is in the 1.0 branch).

It doesn't help that I work purely in computer vision. NLP is a subject I have to teach myself on my own time.
Title: Re: Creating BNX clone for IRC
Post by: rabbit on September 09, 2013, 09:46:02 am
A rudimentary system is to use a Markov chain and just leave the bot idling in popular channels for a while to build the dictionary.
Title: Re: Creating BNX clone for IRC
Post by: nslay on December 22, 2013, 12:51:07 am
One step closer to ShellNotify junk. It now runs with WinMain() instead of main() among other feature and code changes.

Latest version is on SVN, but I'll package a 1.2.0 soon.
Title: Re: Creating BNX clone for IRC
Post by: nslay on December 26, 2013, 01:00:30 pm
OK, tray icon is added. It just needs a picture now! Any suggestions?

Also, I'm not a Win32 programmer. Can someone critique these?

BnxWin32Driver.h (http://sourceforge.net/p/ircbnx/code/HEAD/tree/trunk/BnxWin32Driver.h)
BnxWin32Driver.cpp (http://sourceforge.net/p/ircbnx/code/HEAD/tree/trunk/BnxWin32Driver.cpp)
ircbnx.rc (http://sourceforge.net/p/ircbnx/code/HEAD/tree/trunk/ircbnx.rc)
Resource.h (http://sourceforge.net/p/ircbnx/code/HEAD/tree/trunk/Resource.h)

Thanks.