When is it fast enough?
Still trying to handle 100 million events a day. First we backed up the data source, then we handled the data in the initial stages, but would eventually drop some of the day.

(Part of the system is a queue that holds data until other parts are ready for it. For the non-geeks that care... think of it as a clothes basket. Someone is throwing stuff into it and another person it taking stuff out. If you take it out fast enough, then it is almost always empty. Thats a good thing. You are "processing" the stuff right away. If you start falling behind (it takes you longer to take it out and do something with it than it takes the person to put stuff into it), then you start falling behind. The basket starts filling up. If the person putting clothes in stops, then eventually you'll catch up.)

Tweaked the system some more and got it to where it never dropped data (the basket was never full), but we still weren't keeping up. Looked at what was going on in the code, and for about 150 thousand processed object, we would do 6 BILLION operations. The operations were very fast, but still... you do 6 billion and it adds up.

We figured out a way to perhaps significanly reduce that. I put in the code, we tested it out, and now the system is purring along. Running at about 50%. (In other words, 1/2 the time it is working and the other 1/2 it is just wating). And now we are never getting behind.

It was a cool think. Like I said last month, I love this kind of stuff. So now are system is up and running and we are ready to handle even larger amounts of data. And there are still other ways we could have tweaked the system. But at 50%, we are probably good enough for now... so on to other things.