False positive malware detection from antivirus softwares which could have ruined my business
Short story about my experience with Avast Software products (antivirus Avast with 400+M users and AVG with 200+M users) which started blocking my businesses due to false positive malware detection at the very bad time over holidays.
About me and my businesses
I am Vladimir, 25 years old owner of Czech based Awagon Entertainment s.r.o. company via which me and my colleagues offer complex services for gaming industry. This company operates mainly 2 projects:
GameArter.com. Automated tool for cross-platform management and operation of games. The solution simplifying games development and distribution of cross-platform games, providing services for users and data management, monetisation, distribution, analytics, testing, tracking, hosting and communication with players. Service is designed for indie developers, small and mid gaming companies.
PacoGames.com. Gaming website and social site focused at instant web-based games. Pacogames is a place connecting players with developers where they may communicate and cooperate on improving games. Available for PC and mobile directly in a browser and as PWA.
The story
We run such kind of projects with high dependency at 3rd party products and enormous responsibility to keep it all functional towards developers and players at the same time. Every our mistake could cost developers revenue and bad experience for them and players. We are extremely aware of this and thus, based on all experience we have got from years in the business, we developed relatively robust solution resistant to common problems impending from using the 3rd party solutions. Once any problem is detected, our systems automatically replace nonfunctional service for a different backup solution and inform us about it at the same time to be able to start deal with it immediately.
Our biggest problem today are services that we cannot affect. Which are out of our system. Like Google’s ad system or antivirus softwares.
With Google, we had problem on October 2018, when Google applied backward 50% high deductions of revenue for invalid clicks. This action was very strange, because any analytics we used, neither Google Adsense and DoubleClick panel showed any anomaly at the data or anything. We had all KPIs similar previous months of few last years. In past, we never experienced such problem until the October 2018. In addition to the deduction, there was additional dramatic decrease of eCPM for 66%. Next month December 2018 was the worst month in our history related to revenues. This happened to us and few other gaming companies based on east what I know so far. West-based competition did not experience that and we had to stop being competitive to them for few next months. Since this event I am quite sensitive for events I cannot affect and which may ruin something I have been building for years. 5 Months later, we got a hit from Avast and AVG. ↓
Dealing with Avast and AVG blocking my services because of false positive
In every country, there is a collection of more consecutive days over which people usually do not work, but instead celebrate a holiday. My country is Czechia and one of these days are Easter. If you do not know Easter, then
Easter is a festival and holiday commemorating the resurrection of Jesus from the dead, described in the New Testament as having occurred on the third day after his burial following his crucifixion by the Romans at Calvary c. 30 AD.
Although Czechia is not too Christian land with high percentage of believers, people here use it as a celebration of the coming spring and take advantage of this national holiday lasting 4 days from Friday to Monday to travel or enjoy some adventure. It was same in a case of our company when most of us planned to be off of PC, including me. In fact, we were preparing all for it for a week, to be sure that all works and will work as expected and we will be able to enjoy the few free days.
All started well. It was Friday morning, so all of us started to fill our plans. I personally decided to visit my parents before to go enjoy some fun time to south Moravia, the region of wine. This my plan was destroyed quit soon. Already on Friday afternoon, first messages of following type came to my mailbox.
“Hi, cant play your games from gamearter.com because links are infected and blocked by antivirus see here https://i.imgur.com/nF5HDWB.jpg please do something. I am a website owner who publish your games, and i am worry about my website users.”
Alright, so instead of wine adventure, it was clear, that I will get a headache from a different thing. It’s probably not needed to say, how critical this situation was. Avast and AVG are most used antivirus softwares in the world, they are product of trust. If such program says you that something is dangerous, you logically do not care about relevance and go away as soon as possible. Next time you really think twice, whether to risk and return visit such page.
Due to the holiday, I was the only one with access to a PC in that time. Without deed knowledge of all the complexity of our system.
In this case, first thing I wanted to do was to check history on servers — whether any file was changed in few last days. Here, first shock came — I was not able to connect to the server. This situation in this time, it’s clear. We have been hacked! I immediately contacted support of our hosting solution — VS hosting company — which provides us 24/7 support. In few minutes, we found out, that I cannot join the server not due to hack, but because my static IP address was changed few days ago and thus I am not whitelisted. I was finally able to log in and find out that no file was changed or updated in 2 last days. Also, antivirus program in my PC (Avira) did not had any problem with any of the page on my websites. Same in a case of other 68 antivirus tools in which I tested the “infected” pages via online checker https://www.virustotal.com. Of course, Avast and AVG — the most used antivirus software are not part of the tests. Any reason for it? Probably good point to think about in Avast / AVG. Being part of the checklist would make life of website owners simpler.
Meanwhile, I also started to test all files on server via maldet (LINUX MALWARE DETECT), again without success to find anything. Google search console was clear as well.
Last check I did was manual check of DOM — all loaded files, their domains and redirects. Even in this case all looked ok and it was clear that it is most likely a false positive. So, without any our mistake or infection, we have been just blocked by 2 most used antivirus softwares while losing trust of our end users and clients with every next minute. You really don’t want to experience something like this.
In few next minutes, I managed to report suspicion for false positive to Avast and AVG via their forms (Avast, AVG). For sure, due to seriousness of the situation, I reported it to both also via personal message on Twitter + via public Tweet. On Twitter I found out that Avast blocked also YouTube few days ago, so service of none size is safe.
The bad thing is, that you never know how long it can take until someone takes a look at it so you must start to investigate from what file the false positive could be. Reports itself are not much useful, message HTML:Script-Inf [Susp] may mean anything related to script on the page.
The one more complicating fact was that I did not experienced this problem on GameArter only, but on PacoGames as well. Some time later I got also more detailed message which helped me much.
Because I knew that same file is being used also on different websites which did not experienced such problem, I knew that it is URL, not domain based. So I tried to remove this file from all pages (on PacoGames) where it was easily possible. On GameArter it was more complicated due to the complexity and I managed it 1 day later.
Problem was, that as you can see in the image above, this was only 1 of 2 alerts I received in the same time. From some reason, AVG detected same problem 2 times which puzzled me and cost me expensive time to find another file with a problem, this time with HTML:Script-inf. I was getting such reports also after removing the ad script. Even now, I am not sure why it was stopped after few attempts. In total, my services were being blocked for 36 hours until I managed to detect and remove files with false positive and keep all functional at the same time.
Using Avast and AVG in my PC for investigation
Next day, after waking up fresh to continue my investigation, I used Avast and AVG also for local scan in my PC. Avast was totally useless, actually, all what it found just led to a service requiring paid subscription. In the end, I lost any interest to use the software anymore and switched to AVG which was a good step. Local scan via AVG managed to mark all problematic files although they were not in my PC physically, but in form of certain script only linking to them.
Therefore, if you experience any problem with Avast and AVG, throw away Avast and start investigating with AVG. Both softwares belong to one company — Avast Software, and it is likely that they use same database and detect same threats. AVG will help you more with investigation.
Time to response for Avast and AVG from personal experience
As I mentioned, I reported my suspicion for false positive to Avast and AVG over 3 channels at the same time. Take a note, that time to response is incredibly important, because if you are not able to deal with the problem on your own, you are just blocked till someone else takes a look at your problem.
I have sent all the reports (form, PM, public tweet) around the noon on April 20th.
First reply come from AVG at my PM 2 hours later. This time was quite good, however removing the problem took another 24 hours. On April 22th, 46 hours from the report, I got also reply for the posted form with apologize that false positive warning was removed from my services and should stop be displaying within next 24 hours. Public Tweet remained without reply.
From Avast it was severe ignorance. First reply came to my public tweet in 50 hours from my tweet with general information that all experiencing such issue should use form for reporting false positive suspicion. Reply for my form came next day, with an information that they did not find any warning at my websites. Of course, I had to deal with it on my own in previous 3 days so that I wouldn’t bankrupt due to loss of trust in my clients and users visiting my services. However, needless to say, that after testing on testnet, the pages were not reported even by Avast.
The affection of my business
Firstly, I need to mention the luck that our clients have direct contact for me. There is very low probability, that someone would go over “infected” website to find a contact to be able to report the owner that his website is infected. In such case, our service could be blocked for a long time without knowing it.
Alright, so although we didn’t have anything harmful for our users on our website and in our games, all users and clients using Avast and AVG believe in opposite — a trust software said them that they are in possible risk and did not allow them to access our service. We are able to communicate this with clients by some way, but in a case of users, we do not have any channel to inform them that they were not in risk and therefore they may fear to visit our service again. Really unpleasant situation. Trust is more than money we lost due to the blocking. Right now we are calculating consequences.
All the bad is good for something
This experience showed me another thing I will be extremely careful about. Firstly, I am ending to use all the fantastic antivirus softwares with great results in tests and nearly no performance consumption on background and starting to use AVG and Avast instead. I remember that there were problems with false positive in a case of AVG already before, so it is definitely a good choice to use one of the antivirus softwares AVG / Avast, if you are a website owner. If you are a user, there are better and free antivirus softwares to use, check their big comparation on av-comparatives.org.
Maybe a new service
To prevent this in the future, I decided to study a bit and investigate for options of automatic checks of websites in tens of antivirus softwares. In fact. this is service every website owner should use, because most of us did not track behavior of antivirus softwares at our pages. Doing so could clarify a lot. Because Antivirus is running in a user device and is not distinguishable, there is no way to look at user behavior of users at your website on a level of used antivirus software. You can lose users although you would know it. Without own mistake. I like https://www.virustotal.com service which offers API access. It is definitely something I must look on.
If you are more interested in this topic, check my separated blogpost where I am writing more detailed information for dealing with a threat on a website.
