Machine Learning Weekend: entry task review

The idea to organise a machine learning weekend began to grow organically last year. But a few events actually pushed us into action: Jirka left Seznam.cz and the Python Weekend concept proved itself as a useful means of gathering intermediate-level developers together and helping them to grow further.

After our experience with the Data Challenge, we decided to set up a simple entry task to ensure that only seriously interested people would come.

The entry task involved a blackbox which returned a y-value for every given x-value. The task was to create a mathematical formula that describes the model of the blackbox.

Blackbox code

Simple Flask-based app hosted on Digitalocean.

@api_app.route(“/do_measurement”, methods=[‘GET’])
def do_measurement():
 try:
 x = float(request.args.get(‘x’, 0))
 except:
 return make_error_response(400, “The x-value needs to be a float number.”)
if -2 < x < -1 or 1 < x < 2:
 y = None
 else:
 a_rnd = random.gauss(6, 1)
 b_rnd = random.gauss(5, 0.5)
y = x ** 4–5 * x ** 2 + b_rnd * x — a_rnd
value = {
 ‘y’: y,
 ‘x’: x
 }
 return make_ok_response(data=value)

Expectations

Basically, all received polynomials similar to x⁴ − 5x²+6x − 5 were considered as correct. I also accepted some variance of the absolute member (~ ±1 was ok too) or the linear member due to their random-influenced origins.

Some examples that I evaluated as incorrect:

1. f(x) = x⁴
2. y(x) = -1325 + -3078x + 340846x² + -8573913x³ + 81248879x⁴
3. y=exp(4.03logx)=>y=x4.03
4. y = -3.092e-09x⁵ + 1 x⁴ + 3.301e-05x³− 5.001x² + 4.934x − 5.29

The first example occurred pretty often (in almost 10% of the responses) but I didn’t accept it as a correct one. Values around zero were too important to model it properly — especially in the context of the “Also for some intervals of x, the y-values are missing; feel free to interpolate” sentence. See the graph below. I would accept that if there was a relevant discussion and comparison with other models; but usually there was none. Also, this mistake was often caused by just using the integers as input values for the blackbox (but it often gave good enough results).

Answering with polynomials of higher than the 4th degree was also incorrect because of the random error in the model.

The graph shows how the blackbox behaves around zero x-values.

Another common mistake was when the blackbox was measured only on an interval of positive numbers. I also appreciated calling the same value multiple times in order to model the error for different x-values.

An interesting fact is that there was often a clear boundary between the academic approach and that of the programmers. For example:

res = re.match(r’b\’{“data”: {“y”: (-?[0–9\.]+), “x”: -?[0–9\.]+}}’, text)

This is not how an experienced developer parses JSON. On the other hand, the programmer-made solutions were often missing some important steps like:

• graph analysis around zero x-values
• confident intervals or some kind of error estimations
• trying different models and discussing them

I also checked the readability of the code and if numpy was used reasonably (polyfit function for example). I took into consideration how accessible the formula was for me (as a customer of a bunch of hackers). Your customer usually doesn’t want to install jupyter, explore the comments of your code or even run it to get the results. But I did.

If you want to see an example of a good solution, try for example this one. It is not long and properly made from both development and academic point of view.

If you are interested whether I considered your solution as “good enough”, feel free to use this service http://judicator.pythonanywhere.com/mlweekend/mlweekend@kiwi.com I quickly made for you. Sadly, not all of the good ones were accepted for the event; see below.

Capacity

Unfortunately, Kiwi.com was unable to accept everyone who sent us correct solutions. The tough decision of who to accept for the event was given to our Community Team. They based their judgement on the content of the applicant’s’ assignment form (motivation/background) and my notes from the evaluation.

I suggested to the Community Team that we should repeat the same event for those who were not chosen: wish me/them luck.

Feedback

With 115 received solutions I barely had a few minutes per applicant for the evaluation, so I couldn’t send individual feedback to you or communicate in any way. Feel free to use this discussion on Facebook to provide each other with the Gitlab link to your code and discuss the approaches. I will keep the entry task blackbox service running.