At 12/31/05 07:16 PM, Xiivi wrote: The_Nameless_0ne has 5,550 experience points and no profile.

He should be on your list. Just thought I'd point that out. I see you have a vagrant named "coredamage" in that spot. Thus I assume that's a code name you used for him.

Cool thanks. There were 3 users who never deposited for so long, that I never caught them on the Last 50 Deposits page. So finally I just figured out what spot was missing and how many exp points they had. Then I gave them some random names wondered if anyone would notice. That's 1 down and only 2 to go.

These are the other two:
Rank # 1170
Exp: 4850
Alias: Missing_Alias1

Rank # 1403
Exp: 4380
Alias: Missing_Alias2

I changed them to Missing_Alias until someone spots it and finds out otherwise.

At 12/31/05 03:37 PM, gfoxcook wrote: There's nothing wrong with taking up the reigns again and doing a list that people have done in the past. UNLESS it's going to meet with resistance from NG admins, that is. They decide what's in the best interests of NG. And even though liljim didn't say "shut it down" in his above post, I'm just telling you to tread carefully and if you are going to tool around with programs that may affect NG's performance, it's best to share that with the admins right off the bat, and above all else...

Well I know I'm not affecting NGs performance, as I said in recent threads I might as well not even visit Newgrounds anymore since they'd save more bandwidth by me or you no longer visiting, than what my program does to the site. I just hope that they don't get irrational about this, thinking that somehow my program is affecting it regardless of what I say or what their logs show. ShittyKitty's program brought alarm, when he said that it mines up to 1.2 million profiles in a single day. Well my program doesn't even have to do that, since the "Last 50 Deposits" tells me everything I need to know about Experience Points.

DO NOT SHARE the program with anyone else unless admins say it is okay. Because 1 person using something like that and 10+ people using something like that begin to be exponentially different.

I definitely wasn't planning on sharing it.

ill be on here in march

At 12/31/05 04:16 AM, -Gooch- wrote: I always liked to find out where I stood amongst te rest of the pack, which I do. Now this list is going to be handy for me, since I'll be able to track how many points I need to get past other users who are either inactive or right in front of me. I like this list, and hopefully you'll go into other lists as things progress.

ME to with my daily depositing for over a year I passed so many useres including active one. Will I ever pass Dondoli or Denny probably not.

Thanks much Blue Balls,we needed a list like this. =)

At 12/31/05 02:30 AM, BigBlueBalls wrote: I just thought I'd help out the Newgrounds community with something I am good at. Since I don't have the time to learn Flash, a least I can put together a program in my spare time that you might enjoy.

I want to thank you for making that Top 2000 EXP list. This should be very useful for me when I need to see how much EXP I need to get into the top 1000 EXP and which people are around close to me with EXP. Once again, thanks for that list.

Very cool Bigblue i luv the list and it brings perspective on who the active higher levels are too, which is nice. Also as for the monthly voting top 2000 active apparently over 1000 users were inactive over the last month because i get to vote this time around! Current Exp rank # is 3047.i wonder how close to the cutoff i am.

Well lets start with the obvious I wish you would have told me you were going to do this.
Not that I mind. Yet some sort of notice might of been nice.

I see others have already pointed out the obvious flaws that you have users on this list that will not be able to vote for the monthly voting because they have not desposited in that month yet if they were to show up and vote in that month they could vote in that months awards, and knock who ever is at the bottom out of the voting.
Forexample if 100 uesers in the top 2000 didn't deposit actually the next 100 people would be able to vote for the awards even though they are not in the top 2000exp, since it is activity not exp that qualifies someone.

I am not sure how you got some of the users on the list after reading how you gathered your data, considering I know for a fact some of them have not deposted exp in over a year. I must say it's a interesting way to to gather data. I do agree it does problably does put a lot less load on the server than the data miners.

Well I saw this topic yesterday, but waited, and got a couple questions answered by looking at the site and pay attention to how often you were doing updates. So I'll by pass the silly questions. Since I notice your list auto updates itself after someone deposits. Based on given info are you going to keep this lits as is with the top 2000 exp people no matter if they are active or not?

I'm currently 2,151 but I can still vote monthly. Soon I will be in the top 2,000 if I continue depositing daily.

what a cool idea and you say you will update it every week that really cool...

At 1/1/06 01:05 PM, ramagi wrote: Well lets start with the obvious I wish you would have told me you were going to do this.
Not that I mind. Yet some sort of notice might of been nice.

Well I didn't really think of warning other list makers I'm coming up with this program. I kind of wanted it to be a surprise, but I guess I should have been more considerate of people like yourself who take the time to build them manually.

I see others have already pointed out the obvious flaws that you have users on this list that will not be able to vote for the monthly voting because they have not desposited in that month yet if they were to show up and vote in that month they could vote in that months awards, and knock who ever is at the bottom out of the voting.
Forexample if 100 uesers in the top 2000 didn't deposit actually the next 100 people would be able to vote for the awards even though they are not in the top 2000exp, since it is activity not exp that qualifies someone.

I know, that's where I messed up thinking it was simply just the 2,000 ranking users. As for most active, I'm sure I can calculate that just as easily with the same information I get from the last deposits page.

I am not sure how you got some of the users on the list after reading how you gathered your data, considering I know for a fact some of them have not deposted exp in over a year. I must say it's a interesting way to to gather data. I do agree it does problably does put a lot less load on the server than the data miners.

It took me about a year to build this list, partly by using other lists in the past. Most of the users in the last thousand though were obtained by just tracking the Last 50 Deposits for a long time. People would come out of the woodwork, as liljim pointed out, but instead of having to find this out by going through all 1.4 million possible profiles, I found it much easier by just monitoring that one page. Before it was just kind of a side project for my own personal reference about who was where, then when my list became almost complete, I thought of releasing it to Newgrounds.

Well I saw this topic yesterday, but waited, and got a couple questions answered by looking at the site and pay attention to how often you were doing updates. So I'll by pass the silly questions. Since I notice your list auto updates itself after someone deposits.

Yes it checks the page every few minutes, so it knows when someone deposits. At the end of the day, it reorders itself at the exact same time this site updates, therefore keeping the ranks accurate and the levels are obtained through tracking the Experience Points Required Per Level page.

Based on given info are you going to keep this lits as is with the top 2000 exp people no matter if they are active or not?

I realise now that I was wrong about the top 2000 active users, so I'm thinking of building a more accurate list of the 2000 panel of judges (most active users) I mentioned before, using the same info. Although I'm still going to keep this one going as well.

At 12/31/05 07:24 PM, BigBlueBalls wrote: Ok well I took into consideration of Newgrounds servers into account and I optimized my program so that it's softer than a flea on an elephants back, regarding the bandwidth.

Okay, okay. Enough said, man. Consider me educated now. #;-}>

I didn't fully realise the methods your program uses to comb through and gather data, and I assumed it was loading tons and tons of individual profiles like previous data miners did. I apologise for jumping to conclusions, but I think you can understand why I did so (for one thing, I have very little programming experience myself, and most of it is FORTRAN and matlab and other engineering/math things from college, nothing to do with comp sci/software programming, really).

According to ShittyKitty this was his method "9,000 profiles on Wednesdays, about 90,000 profiles on Sundays, and all 1.2 million possible profiles every month."

Yep. No one had told him there was anything wrong with that, and it's not like he was pulling 1.2 million profiles every day.

But the problem, it turned out, was how RAPIDLY the program itself pulled profiles. For example, if someone was only pulling 1000 profiles or so, but doing so within 1-2 minutes, that could have negative effects on the database server (even if only for that 1-2 minutes once per week and not for 1-2 hours like it takes to do 10s of 1000s of profiles for SK's NGTL).

So anyway, we've all been through that pain in the past, and the nature of the posts I saw from you on page 1 of this topic raised some red flags.

The posts from you on page 2 have lowered those red flags of mine, so my reservations are entirely withdrawn. In fact, I'm glad someone came up with a new way of automating stat collection without impacting NG negatively with mass profile drawing. Using the 50 most recent deposits list is something that listmakers like myself, Newgrundling, ramagi, jonthomson, Dogma, and many others have been doing for 3+ years, but none of us made a program that scans that 50 most recent listing all day every day to catch everyone depositing.

I'm just grabbing one page, "Top 50 Deposits" getting the exp points, their name and ID. Simplest, most efficient, up-to-date and softest way of grabbing the stats. No massive data transfers, just a program that checks one static html page every few minutes and that's it. So it shouldn't even be taxing the database at all, since the html page appears to be updated by a separate program, rather than directly connected to the database.

Again, thanks for the detailed explanations on the inner workings and so forth. No worries here anymore. At least not on that subject. #;-}>

If they see what I'm doing as taxing the server, then I might as well stop coming here because me voting on 5 movies a day certainly is hurting their server much worse than my program.

But then you wouldn't be seeing the ads on the site, either, which keep it afloat, so... heh. #;-}>

That being said, getting the other information (B/P or voting power lists) would have required something along the lines of mining profiles, so that's why I'm not going to bother with it now after reading those threads.

That was part of my worry at the start of this topic, and yep... there's no official daily b/p related listing on NG past the top 50, so you'd have to mine profiles then, and that's not the way to go, unless you want to do manual work or work out some way to do it without mass-profile-loading within short spans of times (if you automated a program to pull one profile per 30 seconds or something, that would be fine. Leave it running all night long and when you wake up in the morning... boom, top 500 b/p list is assembled for ya).

At 1/1/06 05:41 PM, BigBlueBalls wrote:

At 1/1/06 01:05 PM, ramagi wrote: Well lets start with the obvious I wish you would have told me you were going to do this.
Not that I mind. Yet some sort of notice might of been nice.

Well I didn't really think of warning other list makers I'm coming up with this program. I kind of wanted it to be a surprise, but I guess I should have been more considerate of people like yourself who take the time to build them manually.

I don't think she meant you should have warned ALL The other listmakers like myself or others.... I think she meant that because her PARTICULAR listing that she's been doing for years now is the top 500 exp (now top 400) list and your listing is top 2000 exp and your method(s) for collecting the users on it pretty much overlap entirely with hers, and hers doesn't have any users that yours is missing in the top 400 span...

That this list makes her list obsolete... that it would have been polite for you to at least warn her, if not ask for permission (obviously she ain't the boss of you, so just letting her know would have been the thing to do) before unveiling this topic. It just would have been nice to give her a bit of a head's up since your new listing is like a huge comet to her listing's dinosaurness, understand?

For an illustration of my point, here's how she began her 1/1 update of the top 400 exp list on her site (http://ramagi.spasmbot.com/MainMenu.html)

"Well as a surpise I got yesterday with someone doing a top 2000 exp list. Well on that note this may be my last update. Since I checked the list and saw he had gotten everyone on my list. As long it list is going to continue, I see no sense in going on with mine. I'll probably make a final desicion over the next week."

This issue, however, is entirely separate... and far more subjective... than the database issue I brought up at the end of page 1, so... don't take it as connected. I just mention it because I saw ramagi's post and your reply to her.

I'm thinking of building a more accurate list of the 2000 panel of judges

I look forward to the 2000 panel of judges listing. That will truly be a unique listing if you can pull it off... you just need to keep track of the date at which each person in the top... oh, 3000-4000 exp users.... last depositing. And if a user wasn't active during the past month, they don't get into the listing even if they're in the top 2000 exp. etc. etc.

At 1/1/06 10:38 PM, gfoxcook wrote::

It's like a huge comet to her listing's dinosaurness, understand?

lol now that is priceless. I left the rest out due to quote restrictions.
Actually in a way I am old skool, I don't think I'm a dino yet.
Actually now I had time to think about thing, I had made the decided that in the next yeat I was going to either pass off to someone else or was just going to stop. So in way this gives me a nice way to move on with my list.

I like this list it is very useful, I wish he had contacted me long time ago, I could have helped him compile his list easier and posibly faster rather than over a whole year. Considering the data I have compiled over the last couple of years. Plus some aditional info I learned over time.

I see there is another list on the right hand side, showing us our own rank changing. Will that be updated too?

At 1/1/06 11:14 PM, ramagi wrote: At 1/1/06 10:38 PM, gfoxcook wrote::

It's like a huge comet to her listing's dinosaurness, understand?

lol now that is priceless. I left the rest out due to quote restrictions.

Heh.

I was going for a somewhat insane way to phrase it, so I'm glad you enjoyed it. #;-}>

Actually in a way I am old skool, I don't think I'm a dino yet.

I didn't say YOU were. Just your list. And it's true. But that's not a bad thing. I just meant that:

A) your exp list is pretty much the oldest, longest-running, still-surviving listing on this forum. It certainly predates my pentalist, the VP list by me, NGling, and everyone else... and if I still had my top 50 protects list which I started in... March/April of 2003... it would even predate that. And it predates Dogma's EGRL (June 2003).

B) your exp list is somewhat of a dinosaur compared to more "modern" lists that have listed exp. SK's exp list in the NGTL and now this new top 2000 listing by BBB.

That said, I agree with BBB. I enjoy YOUR exp listing. Always have, always will. If you stop, I'll be sad, but I'll get it over it. Still, it's always been fun. I've already checked out the update and it was nifty as always. So thank you for all the great times. #;-}>

One final thought, though... my VP list was pretty much made a dinosaur when SK introduced his weekly top 50,000-or-whatever VP list in the NGTL, but that didn't make me retire the list. I kept it going through the entire NGTL and beyond, and now people STILL update the thing.

So... just a thought.

Actually now I had time to think about thing, I had made the decided that in the next yeat I was going to either pass off to someone else or was just going to stop. So in way this gives me a nice way to move on with my list.

Ah, well... if you'd been looking for a reason to retire the list ANYWAY, then this is a good a time as any, yes. Whatever you decide, I'm sure most of us will back you up on it. #;-}>

I like this list it is very useful, I wish he had contacted me long time ago, I could have helped him compile his list easier and posibly faster rather than over a whole year. Considering the data I have compiled over the last couple of years. Plus some aditional info I learned over time.

It is a nice list, yes. I prefer your list for the top 400/500 portion. But the 500-2000 part of his is certainly nice (I get to see my alt-alt in the top 650, for one thing!)

At 1/1/06 10:38 PM, gfoxcook wrote: I didn't fully realise the methods your program uses to comb through and gather data, and I assumed it was loading tons and tons of individual profiles like previous data miners did. I apologise for jumping to conclusions, but I think you can understand why I did so (for one thing, I have very little programming experience myself, and most of it is FORTRAN and matlab and other engineering/math things from college, nothing to do with comp sci/software programming, really).

I'm glad you warned me about it anyway before I jumped ahead and decided to take on the task of creating other lists that would require bigger data transfers.

The posts from you on page 2 have lowered those red flags of mine, so my reservations are entirely withdrawn. In fact, I'm glad someone came up with a new way of automating stat collection without impacting NG negatively with mass profile drawing. Using the 50 most recent deposits list is something that listmakers like myself, Newgrundling, ramagi, jonthomson, Dogma, and many others have been doing for 3+ years, but none of us made a program that scans that 50 most recent listing all day every day to catch everyone depositing.

Cool, so I'm not so unoriginal after all ;)

That was part of my worry at the start of this topic, and yep... there's no official daily b/p related listing on NG past the top 50, so you'd have to mine profiles then, and that's not the way to go, unless you want to do manual work or work out some way to do it without mass-profile-loading within short spans of times (if you automated a program to pull one profile per 30 seconds or something, that would be fine. Leave it running all night long and when you wake up in the morning... boom, top 500 b/p list is assembled for ya).

For now I'm not going to do anything like that, unless I ask some admins about it. I also won't do it unless I find a way to do the least effect on NGs performance (like I did with this program) and right now I can't think of any other way of doing it. If only they gave us a way to get this info, maybe through something like an RSS feed many news sites have. Then it wouldn't take any massive amount of calls to each profile, but one single call for say a file with all of the stat information dumped in there. Of course I'm sure this won't happen, so I'm guessing we'll just have to wait until liljim has the time to do it. It's too bad though that they don't consider allowing more volunteer work in the programming area, to take time off of liljim's hands. Wade's got volunteers to help him moderate the forums, reviews and stuff like that, while Tom's got the Newgrounds community to help him with his original job of selecting movies worthy of staying in the portal and discarding those that are not. Now liljim, on the other hand, really has no other people here to make his job easier, but I'm sure there's a load of talented programmers here who would love to volunteer our services especially for small jobs deemed insignificant, like stats and stuff like that.

That this list makes her list obsolete... that it would have been polite for you to at least warn her, if not ask for permission (obviously she ain't the boss of you, so just letting her know would have been the thing to do) before unveiling this topic. It just would have been nice to give her a bit of a head's up since your new listing is like a huge comet to her listing's dinosaurness, understand?

I know, I haven't spent a lot of time really getting involved with the Newgrounds community before. There's not one user I've had a conversation with off this site through email or IM, except for a few mods. I do this in all the forums I visit to keep a low profile and keep somewhat disconnected from the community. So I didn't really take anyone into consideration here, when really I should have spent more time discussing this in the Wi/Ht forums and getting to know the other list makers here.

Now I've decided since I released this program, that I'm not going to be so low profile anymore on Newgrounds anyway. So I'll be more communicative about what I plan on doing and things like that.

I look forward to the 2000 panel of judges listing. That will truly be a unique listing if you can pull it off... you just need to keep track of the date at which each person in the top... oh, 3000-4000 exp users.... last depositing. And if a user wasn't active during the past month, they don't get into the listing even if they're in the top 2000 exp. etc. etc.

So I think I have this right... A user who deposits for a month is eligible as long as he has as many or more exp points than the other 2,000 active users right? I'm trying to think about the kind of algorithm I'd use for making this list as accurate as possible. It will be a good list for people who may anticipate whether they are going to be on the list at the end of the month.

To be honest, you must be a bit lifeless to do this...

w00t number494 ^_^

Still a bit pissy that I lost a place. Oh yeah, there are some mistakes in it.

At 1/2/06 03:20 AM, Domnikathor wrote: To be honest, you must be a bit lifeless to do this...

w00t number494 ^_^

After crawling on the floor last night at a New Years Party from too much beer, Rum, weed and Mescal, I certainly wouldn't say that I'm lifeless. Although I felt pretty lifeless this morning when I woke up. It's pretty much all automated, so it updates itself automatically night and day.

At 1/2/06 03:33 AM, Domnikathor wrote: Still a bit pissy that I lost a place. Oh yeah, there are some mistakes in it.

Can you tell me the mistakes? Those numbers are from 4:45am Eastern Standard time last Monday. When it updates again in the morning, it'll be more accurate. Your current ranking in your profile says 494, so it shows that it's in sync with the site's ranking.

At 1/2/06 03:49 AM, BigBlueBalls wrote: Those numbers are from 4:45am Eastern Standard time last Monday.

EDIT: Those numbers on the right are from 4:45am Eastern Standard time last Monday.

I'm a bit confused about Robain's rank. He has 6160 experience points, but he isn't ranked, even in his profile. But will he still be on?

At 1/2/06 03:58 AM, Frostbreath wrote: I'm a bit confused about Robain's rank. He has 6160 experience points, but he isn't ranked, even in his profile. But will he still be on?

That's fucked up. Where did you find this guy? I have no clue why he'd have so many exp points, yet no ranking in his profile. There doesn't appear to be any spot for him there either. All of the users match up to their ranks around where he is supposed to be located. I can't squish him in there or else it would screw up everyone elses rank.

At 1/2/06 04:05 AM, BigBlueBalls wrote: That's fucked up. Where did you find this guy?

We used to talk, but he hasn't showed himself in ages.

I can't squish him in there or else it would screw up everyone elses rank.

That's fully understandable. But hold on to that profile, in case he happens to get himself into a rank.

At 1/2/06 04:29 AM, Frostbreath wrote:

At 1/2/06 04:05 AM, BigBlueBalls wrote: That's fucked up. Where did you find this guy?

We used to talk, but he hasn't showed himself in ages.

Maybe that's why he doesn't have a rank... I'm wondering if Newgrounds takes out your rank after a certain amount of time of inactivity.

That's fully understandable. But hold on to that profile, in case he happens to get himself into a rank.

Alright thanks for the heads up.

Very nice list, almost as sexy as Bedn

At 1/2/06 04:32 AM, BigBlueBalls wrote:
Maybe that's why he doesn't have a rank... I'm wondering if Newgrounds takes out your rank after a certain amount of time of inactivity.

That's fully understandable. But hold on to that profile, in case he happens to get himself into a rank.

Alright thanks for the heads up.

I should warn you there are some others out there like this, I don't recall them all off the top of my head. I never asked the admins about the missing rank, but I do recall seeing a few out there over time. Here is one I thouight of while writing this, afijsohfan.

I'm a bit confused as to how all the information you're pulling is coming from the list of last deposits - because if it were, you'd be missing many people on the list. I've seen at least a couple there that haven't deposited for at least a year...

Beautiful. So if Im constant for about 2 months, I can break 1,000. Thanks Mr blueballs.

liljim
I'm a bit confused as to how all the information you're pulling is coming from the list of last deposits - because if it were, you'd be missing many people on the list. I've seen at least a couple there that haven't deposited for at least a year...

Well, maybe he did a bit of manual hunting too.

At 1/2/06 01:53 PM, liljim wrote: I'm a bit confused as to how all the information you're pulling is coming from the list of last deposits - because if it were, you'd be missing many people on the list. I've seen at least a couple there that haven't deposited for at least a year...

The question was asked by ramagi and I explained how I collected many of the older ones are from other lists or just sometimes dumb luck finding a post from an missing user from my list. This was over the course of about a year and originally I was collecting them for my own curiousity of where I was placed. The "Last 50 Deposits" idea came in later when I was trying to figure out an easy way to catch new people who continue to move up the ranks everyday as my older lists slowly started to become deprecated and full of holes. Over time this method almost completely filled in those holes, except for a couple more I still have lingering.

At 1/2/06 01:00 PM, ramagi wrote: I should warn you there are some others out there like this, I don't recall them all off the top of my head. I never asked the admins about the missing rank, but I do recall seeing a few out there over time. Here is one I thouight of while writing this, afijsohfan.

Ok thanks... maybe if liljim comes back to this thread he can explain why some people have no rank in their profile and don't seem to really fit anywhere in the users list.