How we taught the neural network to identify herpes by photo

Hello, I’m the CEO of Unistory development agency, and I’ll tell you how we created a dermatologist robot and prepared it for launch in the USA and Canada. Iranian businessmen, experienced developers, the Canadian market, open-source ML technologies, six groups of skin diseases.

Rami immigrated to Canada from Iran many years ago. Back in his Homeland, he had a business partner, Mohsen Khoddami M.D., a professional dermatologist. Together, they decided to create an application where a neural network would identify dermatological diagnoses based on photos of problem skin — and they entrusted us with this task. The goal of the application is to help patients worldwide treat skin diseases.

The killer feature of the future application became artificial intelligence capable of providing initial diagnoses based on photos.

Why AI? Firstly, it is the only way to automatically provide a diagnosis based on a photograph of a problematic skin area. The neural network allows automating the process and providing users with basic recommendations. Secondly, using AI as a tool for telemedicine is a powerful PR move for promoting the application.

The concept of the future application

Rami and Dr. Khoddami initially defined the main user scenario:

1. The user with a problematic skin area takes a photo of it and uploads it to the application.
2. Artificial intelligence provides an initial diagnosis — identifying the disease and recommending treatment.
3. Immediately after that, the user can schedule a consultation with a professional dermatologist.

Business partners immediately devised the primary monetization method. Dermatologists who register on the platform and confirm their qualifications will pay a small commission for each paid consultation. They also planned other monetization options: standard in-app advertising and sponsored articles in the “Useful Materials” section.

How We Started Working Together

The task is ambitious — to turn the neural network into a professional diagnostician for skin diseases. With this goal in mind, Rami went to Upwork, and that’s where we, Unistory, entered the picture!

I posted the task on Upwork, and one of the responses came from the Unistory team. The company’s portfolio impressed me: the attention to detail, the advanced technologies, and their experience in the medical field. I immediately felt that these were the right people for us, and I wasn’t mistaken: from that moment until now, everything has been fantastic. Initially, Rami and I weren’t sure if we could bring our idea to life — but Unistory managed it.
Mohsen Khoddami M.D., CEO and co-founder of Dermadex

Our team worked on the project structure and selected the technologies: we decided to use React Native for the mobile application and implement the backend in C#.

Together with the client, we immediately agreed to focus first on finding datasets, selecting the necessary AI model, and training it. Everything else would come later. First, we needed a killer feature!

How we trained artificial intelligence

The first stage of development was the preparation of the Proof of Concept (PoC). We often work with experimental projects, even more frequently with blockchain and AI technologies, so we have a dedicated R&D engineer for creating PoCs. It is they who verify the riskiest technical hypotheses — thus saving money and time on the development of the entire product.

At this stage, we decided that to test hypotheses and develop the Proof of Concept, it would be sufficient to train the neural network to identify six groups of diseases. The next task was to find data (datasets) for training the neural networks.

We explored open-source repositories and found 21 datasets. To our disappointment, there wasn’t as much quality material in them as we had hoped. To address this issue, we decided to rely on zero-shot and few-shot testing methods.

Developers tested several AI models and chose CLIP — the base model that allows for image classification, object detection, and text generation based on images.

We tested the selected models in two stages:

1. Zero-shot: We evaluated the capabilities of the models without pre-training on datasets. This means the model was tested on tasks or datasets it hadn’t encountered before.
2. Fine-tuning (few-shot): Here, the models underwent additional training on our datasets.

The trained neural network achieved a 99% diagnosis accuracy given high-quality photos!

We taught CLIP to identify over 60 different diseases grouped into six major categories:

1. Acne
2. Herpes
3. Eczema
4. Rosacea
5. Psoriasis
6. Vitiligo

The main problem that developers encountered was the shortage of data and the lack of high-quality datasets available openly.

The solution came through data augmentation — we expanded the database by generating synthetic data based on real ones. By introducing minor distortions into the images, we were able to significantly expand the dataset for training the model.

Currently, we continue to search for datasets for training, with the goal of teaching CLIP to work with 30 disease groups.

Preparation for Launch

While our R&D engineer worked on the Proof of Concept and managers prepared the structure, the client handled legal matters. Rami studied what the application should be like to comply with HIPAA standards for the future launch in the United States. HIPAA certification ensures that the product meets security requirements for patient data.

To obtain HIPAA certification, we pre-planned comprehensive logging: every user request must be recorded in the database. Information about all patient, doctor, and administrator actions must be preserved. However, no one should have access to patient data.

Rami is currently preparing to launch the application in Canada. Afterward, we plan to scale into the US and European markets. The launch will proceed step by step, as each country has its own healthcare nuances.

But that’s not all. Rami and Dr. Khoddami are already discussing a separate version of the application for nursing home staff. Elderly people represent a distinct target audience for the dermatological field: limited mobility leads to more skin problems. Together with our clients, we plan to create a custom version of the application tailored to the needs of healthcare workers in nursing homes from the very beginning.

More of our projects on our website → https://unistory.app/en/