Author Topic: Re-writing JBLS  (Read 14714 times)

0 Members and 1 Guest are viewing this topic.

Offline Hdx

  • The Hdx!
  • Full Member
  • ***
  • Posts: 311
  • <3 Java/Cpp/VB/QB
    • View Profile
Re-writing JBLS
« on: June 10, 2008, 08:19:07 pm »
Well, I hate programming for Bnet anymore.
But, I was looking at JBLS's code the other day.
After dusting a few feet of dirt off of it. I realized, the code is really, REALLY fucking ugly.
So god dammen many useless features. Shit that nobody needs to have. JBLS should just run in the background, no interface at all. A daemon.
So, I *may* end up almost completely re-writing it.
Which is where you all come in, Brainstorm time.

Few core features:
Async packet handling:
  -Set number of worker threads for entire server, or set number per connection.
    -Per connection:
      Can cause an ungodly amount of threads [100 connections * 5 per connection ...]
      Would cause no possible cross-connection backup [lag]
    -Per server:
      Low thread count [25 worker threads should be fine, if that]
      Possibility of a queue [lag]
  Personally i'm for the per server route.

Checkrevision:
  -'Intelligent' guessing. AE: packet 0x09 you only get the archive number. Take a guess based on product for the current one in use by battle.net.
  -Multi-threaded, The MAIN lag time of JBLS is in the crev, Namely lockdown. But having 2-3 instances of each crev classes loaded means 3 crevs can be run at once. Once again this could be set num per server, or one instance per connection.
  -Lockdown, Dear god this needs to be improved. we NEED a faster way of loading the files. A more efficient way. JBLS with all files loaded into memory for Lockdown uses 85~mbs of memory. Unacceptable.
  -Inheratance, v1, v2, and v3 of checkrevivion are all exactly the same, cept, v2 introduced padding, v3 introduced 64-bit unsigned ints. Having one core v1 that handles u64s and simply replacing the readFile() function. [thats what its called now] would save a SHITLOAD of room code wise.

Administration:
  I, Will leave that to the outside user. I *might* make JBLS extensible. AE: make helper functions [reloadConfig, setVerByte, blah blah blah] leave others to make there own plugins for it. Oh, hell, I could make the eintire server selfcontained and have things like config files, statistics, etc.. in plugins.

Main thing that pissed me off about working on JBLS back in the day was that I was stupid, I jsut kept adding shit into it. I never really defined the CORE of JBLS and had everything round it. [Though the code I ave now, unreleased, is a bit better at doing that]

So, should I make it plugin sytyle ala JavaOp? This then involves the question of threadding. If a plugin registers that it will handle packet X, and then a client sends 2 X's in a row. The threads would split, 2 would pickup the X packet. Race for the single instance of the handler, And then have to queueup again. Is this big enough of a concern that I should do something about it? How about, when a plugin registers a packet handler, each worker thread createa a subthread. Then again we get into an ungodly ammount of threads.

I know there is a simple way to do this, I jsut am not seeing it.
Also 1/2 this thread was me talking to myself, but feel free to chime in.
As you all know my MAJOR downfall is planning things out to be clean/efficient.
http://img140.exs.cx/img140/6720/hdxnew6lb.gif
09/08/05 - Clan SBs @ USEast
 [19:59:04.000] <DeadHelp> We don't like customers.
 [19:59:05.922] <DeadHelp> They're assholes
 [19:59:08.094] <DeadHelp> And they're never right.

Offline Camel

  • Hero Member
  • *****
  • Posts: 1703
    • View Profile
    • BNU Bot
Re: Re-writing JBLS
« Reply #1 on: June 11, 2008, 02:07:11 am »
Main thing that pissed me off about working on JBLS back in the day was that I was stupid, I jsut kept adding shit into it. I never really defined the CORE of JBLS and had everything round it. [Though the code I ave now, unreleased, is a bit better at doing that]

I promise I won't say "I told you so."

[edit] On a productive note, I'd be interested in contributing to a project like this, but only if there's a source-control system, and I'm a committer :)

I'll draw out some UML for an async server tomorrow at work and post it if I remember.
« Last Edit: June 11, 2008, 02:15:57 am by Camel »

<Camel> i said what what
<Blaze> in the butt
<Camel> you want to do it in my butt?
<Blaze> in my butt
<Camel> let's do it in the butt
<Blaze> Okay!

Offline iago

  • Leader
  • Administrator
  • Hero Member
  • *****
  • Posts: 17914
  • Fnord.
    • View Profile
    • SkullSecurity
Re: Re-writing JBLS
« Reply #2 on: June 11, 2008, 09:30:28 am »
I wrote a pretty nice async socket class for Java that I posted here somewhere. You can use that to listen for connections/data then hand it off to a worker thread.

Offline Hdx

  • The Hdx!
  • Full Member
  • ***
  • Posts: 311
  • <3 Java/Cpp/VB/QB
    • View Profile
Re: Re-writing JBLS
« Reply #3 on: June 11, 2008, 12:38:42 pm »
I promise I won't say "I told you so."
You do realize that JBLS was made like 4 years before you were an itch in sun's groin... Anyways, I was kinda stupid back then. And up until now way to dammen lazy to do anything about it :P

[edit] On a productive note, I'd be interested in contributing to a project like this, but only if there's a source-control system, and I'm a committer :)
Prolly SF or Google Code. Use there SVN servers.
I'll prolly keep the core my-commit only. But have plugins open for you. [I am really leaning twards plugin based]

And ya, Ron I saw your async class a while back when you posted it. I'll look into it more when I get a free moment.

The main thing i'm looking for from anyone isn't code. Its ideas. I bad at making a nice efficient system in my head. So it never comes out right on paper.
~Hdx
« Last Edit: June 11, 2008, 12:42:12 pm by HdxBmx27 »
http://img140.exs.cx/img140/6720/hdxnew6lb.gif
09/08/05 - Clan SBs @ USEast
 [19:59:04.000] <DeadHelp> We don't like customers.
 [19:59:05.922] <DeadHelp> They're assholes
 [19:59:08.094] <DeadHelp> And they're never right.

Offline warz

  • Hero Member
  • *****
  • Posts: 1134
    • View Profile
    • chyea.org
Re: Re-writing JBLS
« Reply #4 on: June 11, 2008, 05:34:19 pm »
Quote
Can cause an ungodly amount of threads [100 connections * 5 per connection ...]

Why so many threads per connection? :X
http://www.chyea.org/ - web based markup debugger

Offline Hdx

  • The Hdx!
  • Full Member
  • ***
  • Posts: 311
  • <3 Java/Cpp/VB/QB
    • View Profile
Re: Re-writing JBLS
« Reply #5 on: June 11, 2008, 07:15:58 pm »
Quote
Can cause an ungodly amount of threads [100 connections * 5 per connection ...]
Why so many threads per connection? :X
Just a random number for worker threads i pulled out of the air.
http://img140.exs.cx/img140/6720/hdxnew6lb.gif
09/08/05 - Clan SBs @ USEast
 [19:59:04.000] <DeadHelp> We don't like customers.
 [19:59:05.922] <DeadHelp> They're assholes
 [19:59:08.094] <DeadHelp> And they're never right.

Offline warz

  • Hero Member
  • *****
  • Posts: 1134
    • View Profile
    • chyea.org
Re: Re-writing JBLS
« Reply #6 on: June 11, 2008, 08:19:39 pm »
Quote
Can cause an ungodly amount of threads [100 connections * 5 per connection ...]
Why so many threads per connection? :X
Just a random number for worker threads i pulled out of the air.

Being serious? Maybe you should have a reason to create a so called 'worker thread' before doing so! I don't see why this can't just be done with like your main server running, and when a user connects they get their own thread. It's not like you need all of these algorithms running at the same time, anyways - after all, the process goes step by step either way.

No reason to rewrite your code if it's just going to continue to revolve around the same poor design principle.
http://www.chyea.org/ - web based markup debugger

Offline Hdx

  • The Hdx!
  • Full Member
  • ***
  • Posts: 311
  • <3 Java/Cpp/VB/QB
    • View Profile
Re: Re-writing JBLS
« Reply #7 on: June 11, 2008, 08:57:07 pm »
Do you not understand the point of this thread?
Its to get a GOOD design set.
95% of the things that use JBLS will not really care for async packet handling. So the way JBLS is now, [single threaded per user req-resp pairs], works fine. But im just thinking of things such as large algorithms [Checkrevision, SRP, not so much SHA] don't impede simple things. [ver byte retrieval, etc..]
Currently, all crevs are queus up in one line. So if 100 people request a crev at the same time.. the last guy is gunna wait a long ass time! [I think it was 200ms for v1, 200ms*100 = 20seconds]
This is all in the planning stage. I don't plan on actually writing anything till at the soonest sunday.
Also, the other main reason I wana do it is the shear uglyness of the code. But unless I have a good design, it'll just end up ugly again.
« Last Edit: June 11, 2008, 09:35:36 pm by HdxBmx27 »
http://img140.exs.cx/img140/6720/hdxnew6lb.gif
09/08/05 - Clan SBs @ USEast
 [19:59:04.000] <DeadHelp> We don't like customers.
 [19:59:05.922] <DeadHelp> They're assholes
 [19:59:08.094] <DeadHelp> And they're never right.

Offline Blaze

  • x86
  • Hero Member
  • *****
  • Posts: 7136
  • Canadian
    • View Profile
    • Maide
Re: Re-writing JBLS
« Reply #8 on: June 11, 2008, 09:10:46 pm »
You mean 20seconds, right?  :D
And like a fool I believed myself, and thought I was somebody else...

Offline Hdx

  • The Hdx!
  • Full Member
  • ***
  • Posts: 311
  • <3 Java/Cpp/VB/QB
    • View Profile
Re: Re-writing JBLS
« Reply #9 on: June 11, 2008, 09:35:20 pm »
Ya, dropped a zero.
Still unacceptable neh?
Even with only 2 worker threads, that's 10s, 4 = 5s, 5 = 4s. I wouldn't go with more then 5.
You all get the point.
http://img140.exs.cx/img140/6720/hdxnew6lb.gif
09/08/05 - Clan SBs @ USEast
 [19:59:04.000] <DeadHelp> We don't like customers.
 [19:59:05.922] <DeadHelp> They're assholes
 [19:59:08.094] <DeadHelp> And they're never right.

Offline iago

  • Leader
  • Administrator
  • Hero Member
  • *****
  • Posts: 17914
  • Fnord.
    • View Profile
    • SkullSecurity
Re: Re-writing JBLS
« Reply #10 on: June 11, 2008, 09:41:51 pm »
I think what warz said is the way to go -- have one worker thread/connection. That's the general way of doing things.

Offline Hdx

  • The Hdx!
  • Full Member
  • ***
  • Posts: 311
  • <3 Java/Cpp/VB/QB
    • View Profile
Re: Re-writing JBLS
« Reply #11 on: June 11, 2008, 09:50:57 pm »
* HdxBmx27 isn't explaining this correctly.
I'm agreeing that 1 worker per connection for the general things is good, Yes. BUT, the thing i'm thinking needs worker threads are the algorithms, CheckRevision mainly.
http://img140.exs.cx/img140/6720/hdxnew6lb.gif
09/08/05 - Clan SBs @ USEast
 [19:59:04.000] <DeadHelp> We don't like customers.
 [19:59:05.922] <DeadHelp> They're assholes
 [19:59:08.094] <DeadHelp> And they're never right.

Offline iago

  • Leader
  • Administrator
  • Hero Member
  • *****
  • Posts: 17914
  • Fnord.
    • View Profile
    • SkullSecurity
Re: Re-writing JBLS
« Reply #12 on: June 11, 2008, 10:05:00 pm »
* HdxBmx27 isn't explaining this correctly.
I'm agreeing that 1 worker per connection for the general things is good, Yes. BUT, the thing i'm thinking needs worker threads are the algorithms, CheckRevision mainly.
I don't think the algorithms need worker threads, the algorithm stuff can be done as part of the connection's thread. People shouldn't be requesting a bunch of things anyways.

Offline MyndFyre

  • Boticulator Extraordinaire
  • x86
  • Hero Member
  • *****
  • Posts: 4540
  • The wait is over.
    • View Profile
    • JinxBot :: the evolution in boticulation
Re: Re-writing JBLS
« Reply #13 on: June 11, 2008, 10:14:53 pm »
What could be done in parallel for a given Crev anyway?  Each result is dependent on the last.

The best performance optimization I could see would be loading all Crev files into memory, which it seems as if you do.  Making it configurable might be appropriate (i.e., giving the user an option as to whether to load a specific game's files into memory).

87mb isn't terribly much these days.  Considering your OS probably pages out mapped-but-unused memory anyway, I don't see this as being unacceptable.  You've got 2gb of virtual process space; you're not even at 5% of that.
I have a programming folder, and I have nothing of value there

Running with Code has a new home!

Our species really annoys me.

Offline Hdx

  • The Hdx!
  • Full Member
  • ***
  • Posts: 311
  • <3 Java/Cpp/VB/QB
    • View Profile
Re: Re-writing JBLS
« Reply #14 on: June 11, 2008, 10:32:41 pm »
I don't think the algorithms need worker threads, the algorithm stuff can be done as part of the connection's thread. People shouldn't be requesting a bunch of things anyways.
People do request a ton of crap. Its jsut them being stupid.
Right now all the 'one run' algorithms are in static form. Meaning One(1) instance of the function is in memory for all threads to use. Unless i'm being retarded and this isnt how it works.
Now the simple thing to do is to make everything non-static, and create/destroy them as needed.

What could be done in parallel for a given Crev anyway?  Each result is dependent on the last.
Whoever said they were dependent on eachother? Only thing crevs use from the last ones are there results in the cache [if its in the cache it doesn't need to be computed]

The best performance optimization I could see would be loading all Crev files into memory, which it seems as if you do.  Making it configurable might be appropriate (i.e., giving the user an option as to whether to load a specific game's files into memory).

87mb isn't terribly much these days.  Considering your OS probably pages out mapped-but-unused memory anyway, I don't see this as being unacceptable.  You've got 2gb of virtual process space; you're not even at 5% of that.
True true, that's one of the main things that bogs it down, file IO time. I don't know, maybe i'm just stingy but I don't like giving it that much.
http://img140.exs.cx/img140/6720/hdxnew6lb.gif
09/08/05 - Clan SBs @ USEast
 [19:59:04.000] <DeadHelp> We don't like customers.
 [19:59:05.922] <DeadHelp> They're assholes
 [19:59:08.094] <DeadHelp> And they're never right.