4 November, 2020
It goes without saying that COVID-19 has taken a huge toll on not just the people all around the world, but on the world economy as well. Once countries started going under full lockdown, industries such as Travel and Tourism, which thrived before the pandemic, has now become one of the crippling industries with a 65% drop in international tourist travel in the first six months of 2020.
Yet there were industries that didn’t just survive, but thrived during this pandemic, namely the Data and Artificial Intelligence Industry.
On 16th of September, 2020, Snowflake, the cloud based data warehousing company, went public and its share prices skyrocketed to $300/share (market cap of $75 billion) at the NYSE making it the biggest IPO in the history of software companies. Palantir Technologies, a big data analytics company, which also went public the same month, has increased its market capital to $17.7 billion. At the time of writing. Datadog, another data analytics company, went public exactly one year ago and its market capital has jumped from $11.7 billion (at the time of IPO) to $29.24 billion, at the time of writing.
What were the extraordinary factors in these companies because of which they didn’t just survive but rose above all the others like a phoenix from the ashes? Well the economic factors will always be at play but apart from that, these companies weren’t just software companies, they were data companies.
1. Popularity of Cloud Data Warehouses
A cloud data warehouse is a database delivered in a public cloud as a managed database for the companies that is optimized for further analytics, scale and ease of use. This service is provided by companies such as – Snowflake, Google BigQuery, Amazon RedShift and many more.
While cloud data warehousing is just a single component of the data pipeline which exists commonly in the industry, companies are investing heavily on this thus making it one of the most important components of the data pipeline
2. Transformation of the Data Pipeline
A data pipeline is the flow and consolidation of data from multiple resources. A data pipeline what traditionally used to be followed by the companies was ETL, i.e., Extract-Transform-Load. With the rise of efficient data warehouses which allows users to store and transform the data, a new pipeline – ELT, i.e., Extract-Load-Transform has emerged. This has also led to emergence of new tools which are better capable of data extraction and analysis, for example DBT, a command line tool that helps users to transform data in their warehouses effectively.
3. Merging of Data Lakes and Data Warehouses
Data lakes are huge repositories of raw data, in a variety of formats, which are low cost but do not support data transactions, whereas data warehouses can store a lot more structured data with transactional capabilities.
Now ‘Lakehouse’ is a new term which signifies the unification of both the terms, i.e. Lakes and Warehouse. Since more companies are moving towards data warehousing, the existing data lake companies are either becoming obsolete or merging to become a data lakehouse. For example, Databricks has made a huge push to pitch itself as a data lakehouse and Snowflake pitches itself to be a potential replacement for the existing data lakes.
4. Trends in Enterprise AI
At one end of the spectrum, most tremendous growth of AI comes from the Big tech (Google, Facebook, Amazon) themselves who contribute a great amount of resources to their R&D and open source tools. Having worked with the big techs themselves, many ex-employees of the company go on to start something of their own. A great example being the former Google employee, Adrien Treuille, who funded his own AI startup, Streamlit.
At the other end, there are those large corporations who have just started to dip their toes into the wide ocean of artificial intelligence, machine learning and data analytics.
In the middle are those companies that had deployed Big Data Infrastructure into their dynamics a long time ago and are now seeing the results of their efforts. Companies like these are now in the deployment phase where they can now embed machine learning and artificial intelligence in their production and a variety of business applications. For example- the pharmaceutical industry.
The scale at which adoption and deployment of these technologies have taken place in the companies has been tremendous. This just shows that humans with their tendency to adapt to the newest of environments and technologies have yet again amazed us. We aren’t the ones who resist change, we would rather embrace it.