The reason being that it was another example to me of a post that was asking the wrong question of our industry and making broad generalizations that, from my experience, are not true when it comes to the convergence of the big data and security ecosystems. The question being asked was this:
What is a bigger challenge for large scale security data analysis efforts, Scalable Platform or Effective Detection Content?
Two points to consider:
- I am getting really tired of those people that are saying that building a security data lake is a fools errand. You can think that it is just a waste of time to do this, but the reality is that those that are doing it today, for the most part, are forward thinking organizations that have a strategic vision and plan for how they are going to use the data in the near term. I have personally visited with 50+ companies in the last 2 years that all have a vision for how they are going to use the data they are collecting. And even more, most of those companies that are building these lakes, are holding this data already anyway for compliance or regulatory purposes and building a lake simply makes that a cheaper proposition for them. Seems pretty smart to me
- The greater challenge that the larger big data ecosystem is having is the complete lack of true applications that sit on top of the stack to provide the next layer of value. Cost savings is the first layer of value that these companies are getting from building a lake for security data. The other is value that comes from leveraging pre built apps that can take advantage of this architecture. Unfortunaltely, these apps are not coming as fast as the customer base deserves and needs, but they are coming. The reality is that building the platform is not easy, in fact, it is very challenging. But so is building the analytical apps on top of it. It is not about just writing some python code to build models. It is about building code that not only analyzes data, but does it at scale and can do it in a multi-tenant processing environment. And if you think this is easy, then you really don’t know what you are talking about.
Again, I think the question asked by the blogger, is really a naive one. It is not about whether or not one is more challenging than the other. It is a question of maturity of the offerings in the market and how they are delivered. Because both building a security data lake and building the app on top that can provide analytical value are ridiculously hard/challenging. Our job as a community is to do our best to hide this complexity and provide a software package that is easy to consume and get value from. Not argue about which part is more challenging.
