Big Data and Machine Learning: Two peas in a pod
Take a step back and think about the connection between big data and machine learning. To really leverage big data you can use the automated analytical capabilities of machine learning. And machine learning algorithms increase in effectiveness when the size of the training dataset grows. And so big data and machine learning techniques go together.
Big data can use the network to aggregate distributed data in the cloud. Machine learning can then leverage the collected data to generate insights.
I have several clients who request big data features within machine learning projects. It is the rule, rather than the exception. For example, scalable web scraper technology can aggregate data (the big data part) for a deep learning model to analyze (the machine learning part). Another example I see often is distributed database requirements and communication between cloud instances (inside a cloud network) combined with distributed GPU-accelerated data analysis models like image recognition and text classification. These machine learning models can be distributed because they do not learn once they are deployed. This architecture is further supported by a geographically distributed content distribution network (CDN).
Big Data and machine learning further intersect in the area of mixed-initiative systems. For example, a recommender system that mines live tweets can involve collaborative filtering that uses big data for the collection and AI for the decision making. There are several obvious cases where a bot and big data mix together to serve some semi-autonomous or even self-directed activity.
What does this mean to you?
Well, assume every machine learning or AI project is also a big data project. Your team is not throwing big pieces of big data software into the project for fun. And in the same scenario I’m not trying to up-sell you on extra services when big data items are added to our proposed statement of work (SOW). Machine learning and big data really do go together.
Think about it.
Happy coding!
-Daniel
daniel@lemay.ai ← Say hi.
Lemay.ai
1(855)LEMAY-AI
Other articles you may enjoy:
