Our AI Hackathon Journey

Published in

CyberArk Engineering

10 min read3 days ago

The latest advancements in AI technology introduced a huge change to the software industry. Developers today can harness the power of AI and build things we could only dream of before.

This excitement hasn’t skipped the development group I’m part of. We were anxious to find ways to enrich our product with AI powered features and dazzle our customers. We had a few ideas and some were even solid enough to be prioritized by our product manager. Overall, however, things were moving slow. Too slow. We wanted to speed up and show some results — so we thought: How about a hackathon?

The Challenge

Packed with energy, we were eager to get going. There was just one small problem: We didn’t have any real knowledge, let alone experience, building AI- based features. Hackathons are all about squeezing what normally would take weeks into a few days by moving fast, cutting corners, and focusing on the core issues. How do you do that when you don’t have the background to support the decisions you need to take?

Setting our Goals

To fill the gap between our desires and the resources at hand we decided to begin with what we aim to achieve. We started by analyzing our situation and arriving at two main conclusions:

This is not a “classic” hackathon — we can’t just propose projects and hack all night, it won’t work.
We have a lot of unknowns: lack of knowledge, lack of tools, limited tech understanding. The bottom line is that we don’t know how to solve problems using AI.

At this point you might be thinking — this is a pretty big gap they have here. They should just forget about the hackathon and find another way. Well, not so fast. We saw it as a challenge. And we love a good challenge.

The way we saw it, all those things we don’t have are not showstoppers, these are potential goals for the hackathon. Instead of focusing on building prototypes, we’ll focus on learning. This time, the prototypes will be more of a tool than a target.

After some discussions, we defined the following three goals:

Learn and experience. We wanted to leverage the hackathon to learn as much as we can. This can include experimenting with tools and libraries or taking online courses on platforms such as LinkedIn or Udemy.
This is the main goal because acquiring relevant knowledge is key for anything we’d like to achieve in the field of AI in the future. Even if we fail to deliver working prototypes in the scope of the hackathon, we’ll be able to use the experience we gain later.
Mindset shift. It’s a new world and we need to adapt, including how we handle data. We’re used to thinking of it as a functional resource: We save only what we currently need and discard what we don’t the moment it’s no longer required. But data is a resource on its own and we need to treat it accordingly. We can analyze it and learn things about the way our system works and how our customers are using it. This affects what we collect, how we retain it, etc.
This is just one example. In general we want everyone in our group to start thinking differently after the hackathon about what we can do with AI, and what we need for it.
Proof of concept. We focus on learning, but we still want to show that our ideas could indeed be implemented using AI. A major part of the learning is experimenting, so we just need to find a way to combine the experiments into our product and produce some prototype.

With our goals set it’s time to move to the next, and most important step — preparations.

Preparations

Hackathons typically span over no more than a few days, so you want to focus the time on actual development to maximize results. Do to that, anything that can be done in advance, should be. This includes process aspects such as project definition, project scope and team members assignment. It should also include technical preparations like solution design, environment setup and testing of tools or services that are part of the solution. Ultimately, the purpose of these steps is to minimize the risk of blockers popping up in the middle of the hackathon, which could result in failure.

Our case was no different, just more complex. We still had to prepare, but we had to do it in a way that addressed our unique needs, which were:

Feasibility analysis. Excited as we were about our ideas, we needed to verify they are relevant for AI and doable in the scope of a hackathon. To do that we had to understand what type of solution will be suitable for each project — generative AI? Machine learning? Maybe something else?
Required tools. We had to figure out how to implement the solution. There’s a variety of options to choose from wherever you look. If it’s an ML project — which algorithm to use? And which library? Or maybe use a service? (We’re on AWS and they have a service for almost anything). If it’s Gen-AI, should you go with chat-GPT? Maybe Claude or Llama?
Our familiarity with all the above was very superficial. To choose the direction for each project we had to extend our knowledge at least to a point by which we can make these decisions.
Data analysis. Every AI-based solution requires data. We had to understand what data we need, the quantities, and how diverse it should be. (Data is a key aspect of any AI feature — we’ll discuss it in detail later).

To increase our chances of success we enlisted the help of selected AI experts from another group in the company. They joined us as advisors, mainly in the preparation phase, helping us shape the solutions and pick the proper tools.

Since the preparation phase was so important we had to be organized and verify nothing was missed. We defined the following iterative process to manage it:

Kickoff meeting. Held about 3 weeks before the hackathon. The main purpose of this meeting was to align all team members. In this meeting we presented the project and discussed its scope and data needs.
Prep meeting. In which the team started raising solution options, devised the DoD (Definition of Done) and reviewed the available data.
Setup. Used mainly to learn and prepare the data required for the development and testing of the project.

Additional prep meetings were held when needed to review the setup progress. Additional questions were naturally raised as the teams dove into the details. We used the prep meetings to discuss and close any open issues that surfaced. Overall, the setup steps spread across two weeks, with a total of 2–3 prep meetings and roughly 10% of the time allocated for hackathon related tasks.

Looking at the preparation process illustration we can see a recurring theme — data. It appears in each step in the process, and it shouldn’t come as a surprise since data is an integral part of AI. Machine learning models need data to train on. Generative AI models need data to provide answers. We already mentioned the importance of data earlier, let’s discuss it now in detail.

Problems with Data

As we started analyzing our data needs, we found two main problems that surfaced in more than one project.

Complex data sets

For some projects where we planned to use machine learning we noticed that our data was too complex. Training a model is not a simple task. Data scientists can work weeks, even months, shaping and adjusting the data to get good results. We had only three days. We decided the best solution would be to reduce the data model to something small and simple enough we can manage in a short time span. It’s easier to explain with an example.

Example: Electronic Appliances System

Say our system manages electric appliances, and we want to add a feature that finds similar items to catalogue them accordingly. These patterns should be able to include any of the appliance properties. However, the appliance object doesn’t have a well-defined structure. Apart from a few properties, most of the object is flexible.

Here’s an example of an appliance object representing a TV:

{
  "type": "TV",
  "brand": "LG",
  "model": "xg249d",
  "attributes": {
    "size": "55 inch",
    "smartTV": true,
    "wifiSupport": true
  }
}

type, brand and model are very basic and relevant for any type of appliance, but the data under attributes will be quite different for a fridge or washing machine. So, the challenge is: how do we find patterns when we don’t have a clear structure?

First, let’s break it down to more specific requirements for our model:

Support mandatory properties with a predefined set of possible values (e.g., type, brand).
Support mandatory free text properties (e.g., model)
Support optional properties (e.g, anything under the attributes object)

Type 1 properties are simple. We know they always have a value and it’s bound, so the model only need to be able to support the equals condition (e.g. type equals “TV”, brand equals “LG”.

Type 2 properties are more complex. They can’t be empty, but they’re not bound so we need to support additional conditions such as contains and startsWith (e.g. model startsWith “xg”).

Type 3 properties are the hardest to handle since we don’t know which properties a specific object will have, or even what type of values they’ll hold. This means the model can’t assume anything in advance.

So, how can we reduce the appliance object and make it simpler?

The Type 3 properties are the biggest challenge — we’ll ignore them completely.
For the Type 2 properties, we’ll support only the startsWith and equals condition.
Type 1 properties don’t require any reduction.

Now our model only needs to be familiar with a minimized, simpler version of the appliance object. The patterns it’ll find will be simpler and won’t answer all needs, but that’s OK. Remember, our goal is to prove we can solve this type of problem. Extending a simple model to a complex one is like starting with a beta version and adding features later.

Missing data

Another problem we faced was that for most projects we didn’t have real data available. The solution here is to synthesize data. It’s simple in essence, but we must make it as similar to real data as possible, otherwise the model won’t behave as expected when meeting the real data.

Some recommendations:

Find the proper balance. The synthesized data can’t be too random, but don’t make it too structured either. It’s OK to randomize certain values, but the overall data should be generated in a way that properly represents real data. Beware of overfitting*.
Be familiar with your data. Understand what each property represents. What type of values they hold, dependencies, etc.

* Overfitting is an undesirable machine learning behavior that occurs when the machine learning model gives accurate predictions for training data but not for new data.
Source: Amazon

Let’s go back to the electric appliance example and see how we can simulate a data set. Since we expect to find patterns in the data, we must include some in the synthesized data set.

Here’s one possible sample:

Type, brand. Both have bound values, so we just pick a few.
Model. This can basically be anything, however this is where the familiarity with your data comes in. If you look at the model value of a real appliance it’ll appear random, but those letters and numbers usually have meaning (size, generation, etc’). With that prior knowledge, we can assume the model value is a good candidate for pattern matching based on subsets of the string values. Note that the actual value isn’t important when generating the data, as long as we introduce patterns the engine can find. In this example I created random values where you can see hints of patterns in length and location of specific characters.

With this generated data set we can test our ML model.

Our Achievements

With the rigorous preparation phase behind us, we were ready for the hackathon itself, and I bet you’re eager to know how it went.

Let’s quickly summarize by going back to our original hackathon goals and see how we did:

Learn and experience. The teams learned about various concepts, libraries and tools from the machine learning and generative AI worlds. Four projects used machine learning models, the fifth used Gen-AI. Three projects synthesized data sets to train their machine learning models. This goal was definitely achieved.
Proof of concept. We had five projects in total, four of which achieved a working prototype. This was better than expected.
Mindset shift. This is a tricky one as the results can only be tested in the long run. Maybe I’ll write about it in the future :)

At the time I’m writing this one of the projects has already been prioritized and is currently being developed, aimed to reach the first customers in a few months. In addition, the project that failed to reach a working prototype received additional time allocation and showed promising results. It’s also expected to be prioritized in the near future.

Putting it all together, we can be very proud of our achievements.

The Lessons We Learned

Jumping into the AI development world can be overwhelming. With so many possibilities to explore and different paths to take it’s hard to know where to start. I hope that our story gave you some confidence and maybe sparked the motivation to do your own AI hackathon.

I’ll end with the top lessons we learned:

Allow enough time for preparations. It’s time well spent!
Don’t be afraid to explore other angles. Some ideas might prove to be too challenging. Try simplifying them, maybe looking at them from a different angle.
Invest in your data needs analysis.
If you generate data, be careful with the assumptions you make. Don’t guess, make sure to rely on facts and remember you’ll need to validate your assumption once you have real data.