Forefront of Data 03

How to compare data & analytics vendors | Databricks State of Data & AI 2023 | Analytics strategy template | Analytics for the Art sector

Jun 04, 2023

1. Databricks report on the State of Data + AI 2023

Databricks released their 2023 State of Data & AI report last week. Obviously we have to keep in mind that this is a vendor and that their report is based on input from Databricks users only, but I found it a really interesting read and it’s a great looking report too - I would recommend taking a look.

In summary, the report talks about the rapid acceleration of the use of NLP, which has been opening up use cases across businesses. Linked to this, the number of companies using SaaS LLM APIs (e.g. to access ChatGPT) has grown 1310% between the end of November 2022 and the beginning of May 2023.

In regards to data & AI products, Databricks users are mainly using products based on open source and they reported that Microsoft Power BI was the most popular data and AI product to be used on top of their Lakehouse platform through 2023. The second most used product was Plotly, the low-code charting platform aimed at data science teams.

The report also talks about the productionisation of ML models and that 1 in 3 models experimented with ended up in production, in comparison to 1 in 5 the previous year. This makes sense - the building and deployment of machine learning continues to mature.

Finally they call out the accelerated growth in the data integration market; tools enabling the integration of many data sources from across a business into one consolidated view. They report a 117% YoY increased in adoption in these types of tools, and this reflects my own experience with my clients - it’s a hot topic.

Thanks to Databricks for publishing the report - well worth a read.

2. Factors to consider when comparing Data & Analytics vendors

As a Head of Analytics, choosing the right data and analytics platform will be a key activity in your roadmap, and for members of the team it will also be an activity you may well help with and as a minimum you will certainly be impacted by the outcome.

What are some of the key factors to consider when assessing vendor options? The detail will depend on the type of platform you’re on the market for, for example an all in one data & analytics platform or a data warehouse, but regardless, the factors below are typically key.

A primary consideration when comparing data and analytics platforms is scalability and performance. The platform should be able to handle the volume, variety, and velocity of data that your organisation generates. Consider the platform's ability to process and analyse large transactional datasets efficiently, as well as handling other types of data such as log files and documents. Additionally, evaluate the platform's response times, throughput, and overall performance.

Data integration and connectivity are crucial for larger organisations to consider for an analytics platform. A typical D&A team will be interrogating data from across the business: ERP systems, operational systems, email servers, CRMs, etc. The ability to quickly connect and ingest data from different types of systems is typically a core requirement. Look for features like data connectors, APIs, and support for popular data formats to ensure seamless integration with your existing data ecosystem.

Of course, the platform's core purpose will be to derive meaningful insights from data. Therefore assess the platform's analytics capabilities, including data exploration, descriptive statistics, predictive modelling, and ability to build and deploy algorithms. Furthermore, evaluate the platform's visualisation capabilities - what is built-in and is there easy integration to solutions like PowerBI and Tableau.

Data security and compliance are critical factors to consider. Ensure the platform adheres to industry-standard security protocols and offers robust data encryption, has access controls and authentication mechanisms. Also consider the platform's ability to track and audit data access and maintain data privacy throughout the analytics process.

Is the platform easy to use for technical data analysts and data scientists, and does it enable a good user experience for the less technical business users who may be consuming insight?

Consider the platform’s community, support and roadmap. When you’re using this platform how accessible will help be, whether it is the vendor itself or other users of the platform online, does the vendor have a good pipeline of future enhancements to ensure the technology continues to improve?

Finally, of course the price is another factor to consider! For a platform in this area, you will likely be considering license fees, implementation fee/effort and support. You will also be considering the impact on the time of your team and the wider business & IT.

3. Insight from the Gartner Data & Analytics Summit 2023

Garter’s D&A Summit is somewhat of an exclusive event, with ticket prices at nearly £4,000 per person and, as far as I’m aware, no cheaper options for freelancers or more junior staff.

I try to get there most years (a perk of the corporate job) and I do recommend trying to go in future years if you can secure the budget - it’s the place to sense-check the vendor landscape (platforms, tools, etc), get some inspiration on longer-term strategy, and hear some real-life case studies.

This year’s summit in London was last week and I couldn’t attend as I had a trip to the Middle East, but I wanted to share some other people’s reflections on the event and the key takeaways.

Jamie Gordon of Datactics talks in his brief post about the emphasis on embedding a D&A strategy as part of an organisations DNA and the need to quantify and communicate the impact of any D&A initiatives.

Finally, Malcolm Hawker of Profisee shared his reflections on the Orlando version of the summit (from a couple of months ago) here. Malcolm refers to presentations covering the latest frameworks such as data fabric, data observability and composability, and guidance from Gartner that analytics platform can’t currently do all of these in a single piece of technology. He makes a great point that many organisations will have found this interesting but are way off the maturity level to properly think about these frameworks and purchase multiple pieces of technology. But interesting insight to help them shape their longer-term vision, perhaps. Malcolm also mentions that GenAI, as expected, was a common theme and some of the big vendors like Microsoft presented on how GenAI was being built into all of their D&A products.

Other insights & useful links

Strategy - Data and Analytics Strategy Template. Link

Article - The art of data: Empowering art institutions with data and analytics. Link

Case study - Travis Perkins Plc: Using a data-driven approach to improve sustainability and customer focus. Link

Article - Forbes: The Democratization Of Business Data Analytics. Link

Article - ChatGPT: The End of Data Analytics as We Know It? Link

Tech - How Tableau GPT and Tableau Pulse are reimagining the data experience. Link

Handpicked job opportunities

UK (London) - Senior BI Analyst, Lightricks

UK (London) - Staff Data Engineer, BP

UK (London) - Head of Data Science, Monzo

US (Washington) - Data Analyst, UNHCR

US (NY) - VP of Data & Analytics, LVMH Perfumes & Cosmetics

US (CA) - Data Scientist, Mercedes-Benz

Thanks for reading, and see you next week.

Topics covered by Forefront of Data

Tools, Technologies & Approaches Database Tech | Analytic platforms | GenAI | ML | Coding Languages

Strategy Implementations | R&D | Prioritisation | Team Structure

Life as an Analytics Specialist Career | Training | Productivity | Hacks | Polls