My Blog

DataOps, a portmanteau of "data" and "operations," is a set of practices, principles, and tools aimed at streamlining and automating data operations, ensuring the rapid and efficient delivery of high-quality data to end-users. It draws inspiration from DevOps, which emphasises collaboration between development and IT operations to accelerate software delivery and improve its quality. DataOps extends this collaboration concept to data-related processes. It involves cross-functional teams of data engineers, data scientists, data analysts, and other stakeholders working together to enhance data pipelines, reduce latency, and ensure data reliability.

As I am building up my data engineering team, I am adopting many of the things I have learned from building DevSecOps teams. Learning from application development and support, what matters these days are "Speed and Agility". With DataOps, just like DevsecOps, I can build or change data pipelines to respond quickly to changing business needs and market demands. By automating data pipelines and reducing manual interventions, data can flow seamlessly from source to destination, allowing for faster decision-making.

Unique to data, is the concept of "Data Quality". High-quality data is crucial for accurate analytics and business insights. DataOps practices can ensure that data is cleansed, transformed, and validated before it reaches its intended destination, this is not unlike automated code testing in DevSecOps structures.

Next we have "Collaboration". DataOps fosters collaboration between different teams within an organisation. Cross-functional teams can work together to define data requirements, build data pipelines, and address data-related issues promptly. This is probably harder than DevSecOps team as many users of data are not familiar with the IT world and would need a lot of help to onboard into the DataOps teams.


Lastly, we have the concept of "Risk Mitigation". By automating data operations and implementing robust testing and monitoring processes, DataOps helps mitigate the risk of data breaches, errors, and downtime. Prior to the implementation of DataOps, these functions were very ad-hoc in nature and can cause significant delays in decision making.

To successfully implement a DataOps team, there were a few learnings:

1. Collaboration: Foster collaboration between different teams involved in data operations, breaking down silos and promoting knowledge sharing.
2. Automation: Automate repetitive tasks, such as data ingestion, transformation, and testing, to reduce manual intervention and increase efficiency.
3. Continuous Integration and Continuous Delivery (CI/CD): Apply CI/CD principles to data pipelines, allowing for continuous integration of new data sources and continuous delivery of high-quality data.
4. Version Control: Implement version control for data artefacts and pipelines to track changes and enable rollbacks when needed.
5. Monitoring and Feedback: Establish robust monitoring and alerting systems to proactively identify and address issues in data pipelines.
6. Data Governance: Define and enforce data governance policies to ensure data accuracy, security, and compliance with regulations.

The hardest part was putting together a robust Data governance structure and framework. Many organisation are not mature in this area and it would take time to identify the right resources, educate the teams, and enforce the discipline needed to govern the data lifecycle.

DataOps represents a transformative approach to data management, aligning data operations with the principles of speed, quality, collaboration, and automation. As organisations continue to grapple with the challenges of managing and leveraging their data effectively, embracing DataOps can be a game-changer. By streamlining data operations and fostering collaboration among diverse teams, DataOps paves the way for data-driven decision-making, enabling businesses to stay agile and competitive in today's data-driven world.

There is a lot to learn from history. Cyber warfare in many ways reflect old fashion physical warfare, so one of the things I tend to look at is how we can learn from previous generations in this aspect. In this post, I took a look at the evolution of naval warships and how they have evolved over the years and got to today's modern design.

Instead of writing a long text, I made a video to show some of these points and how ships today are built with an "assume breach" mindset and some of the lessons learnt in the implementation.

In today's world, we are getting more and more connected and thus many things in our life are made up of complex systems. When we look at how complex these systems of systems are, we can easily see that the way we have been taught to "isolate" problems and work on it, sometimes can have outcomes we did not want.

I have been actively teaching system thinking classes and I wonder why this is not more widespread in our education systems. Simple tools like the causal loop diagramming technique and the iceberg models are great toools to equip everyone with. We need to educate more people to look beyond the obvious and go deep when troubleshooting problems. I have added a short video i made on the iceberg model which I find simple and easy to adopt.

Since 2018, I have been creating classes on Quantum Computing. Started out with basic IBM Q Composer workshop (drag and drop gates visually) to the current 2-day class I conduct with Dr Lua Ruiping (my partner in crime :D ). The can person class has a lot of labs and examples we build using Qiskit on IBM Q and explores various aspect of quantum computing application like codify Shor's algorithm to crack encryptions using factors of 2 prime numbers (e.g. RSA).

Today I decided to see if I can make the class into short videos and here is chapter 1 in the new series:

As we get deeper into the architecture series, I would like to take the opportunity to cover a bit more on the theoretical aspects on why we model and why we need to follow methods or framework. I put it similar to learning how to drive - you need to learn the highway code for every country (or environment) you are going to drive in, the local laws and also maps. Those are the models of the environment (or system).

How you drove when you were taking your driving test and how you drive today would certainly be very different. When you were taking you test, you went through a "checklist" of behaviour like looking at all the mirrors, making sure your indicators are on, confirming your blind spots by turning your head, etc. before you change lanes. but today, based on your experience, you might do it all at once or take short cuts or rely on blindspot sensors in your new car.

The point is, you might not follow the "checklist" of learning how to drive, but you certainly remember the safety aspects and why you do it, although might be more subconsciously.

hope you enjoy this session.

Following on to my earlier post on IT Architecture, this is a follow up post on the types of architecture in IT and their relationships.

This is only 1 simplified view, there are many types of architecture that are not reflected here.

As I try to find the most common types of architecture to illustrate the eco-system, so remember that context is important, different industries and domains might require specialised sub-domains.

I hope those embarking on the journey of learning will find this useful.