Marcin Pionnier on finishing 5th in the RTA competition
I graduated on Warsaw University of Technology with master thesis about text mining topic (intelligent web crawling methods). I work for Polish IT consulting company (Sollers Consulting), where I develop and design various insurance industry related stuff, (one of them is insurance fraud detection platform). From time to time I try to compete in data mining contests (Netflix, competitions on Kaggle and tunedit.org) — from my perspective it is a very good way to get real data mining experience.
What I tried
As far as I remember, the basis of the solution I defined at the very beginning: to create separate predictors for each individual loop and time interval. So my solution required me to build 61x10=610 regression models. I was playing with various regression algorithms, but quickly chose linear regression — because the results were good and the computation time was short. I think the key to get quite good result (especially on public RMSE 🙂 ) was the set of attributes used. I used the following attributes for the linear regression for each individual loop&time interval:
— number of minutes from 0:00 hours up to current moment (“now”)
— average drive time for given loop&interval
— loop times for current moment and some number of historical moments
before (the number of time points and the loop varied between the
methods)
— differences between “neighboring” time moments for the above data:
just differences or differences transformed with logistic function
(1/1+e^-difference). Use of logistic function gave a jump from public
RMSE at about 198 to 189. The idea to use of sigmoid function here was
just my intuition inspired by differences distribution.
— “saturations” for for each loop (except the 2 first loops at both
directions ).
I introduced the simple (and very naive) model of traffic growth:
If the speed at given loop is up to 40 km/h — the saturation is 1;
If the difference between the previous loop and the given loop is more than 5 km/h: it is assumed that this road part is partially saturated: there is segment that is moving at 30 km/h and second segment with the same speed as in the loop that is before given loop. The saturation is derived as the proportion of first segment to the whole road part. Each loop detector has its minimal value in RTAData file — after the regression this minimal value was used if predicted value was less than minimum.
I did not use historical data at all — I found them useless during the initial tests (maybe too hastily). The only source of data for learning and testing was RTAData and lengths files (also no weekends, holidays, weather conditions).
What ended up working
For each of 610 regression models the following 3 models were competing. Models were being trained with all data availabe in RTAData
file:
Model 1: For all (61) loops: current + 5 times moments before and 5 simple differences — 675 attributes,
Model 2: For 10 before, current and next 9 loops (if available or less): current + 9 times moments before and 9 simple differences, saturations (for current time moment only) — 204 to 404 atrributes,
Model 3: For 10 before, current and next 9 loops (if available or less): current + 9 times moments before and 9 sigmoided differences,
saturations (for current time moment only) 204 to 404 atrributes,
Model with least RMSE computed on the train file was selected for particular loop. It is not a very good strategy, however I thought
that generally linear regression was resistant to overfitting (it is not true — as the number of variable grows, the more variance can be explained — this is what I have learnt).
This strategy gave me public RMSE 189.3
I added also 4th model, that I just used for 15, 30 minutes predictions arbitrarily:
Model 4: For all (61) loops: current + 5 times moments before and 5 sigmoided differences, saturations (for current time moment only) — 614 attributes. This turn gave mi 188.6 public result.
What is interesting, the best private solution (however not selected by me since I relied to much on public results) was 190.819 (public 197.979) , it was just the model 3 described above combined with model 5 (model 5 was used for 15,30,45,60,90 minutes predictions arbitrarily, rest model 3):
Model 5: like model 3 but also loop times are “sigmoided” not only differences.
What tools I used
My solution is written as Java application with Weka linked as library (as always when I try to compete in data mining contests). Since linear regression requires to solve matrix equation (in this case quite huge), the memory allocated by the program was becoming more and more important issue (3,5GB for one thread) — at the of the competition i was using computer with 4 processors and 12 GB of RAM — with 3 separate threads building and testing the models. The whole computation for my last attempts took about 48 hours of computations.
Originally published at blog.kaggle.com on February 17, 2011.
