Automate Your Data Governance

Scott Knight — Tue, 10 Oct 2023 00:00:00 +0000

After reading about 23andMe’s recent data breach issue this morning, I’d like to share some thoughts on data governance from my perspective as a Data Automation Engineer.

Data drives just about every aspect of our world, and increasingly so as technology continues to evolve. A growing challenge with this is the art of managing and governing this ever-growing volume of data in a way that ensures the proper maintenance of its quality and security.

In my career as a data professional I have seen how data governance can slip into an afterthought, particularly in small or mid-sized organizations. Luckily, automated data governance can be a powerful tool to ensure data quality, security, and compliance without much effort.

Let’s take a moment to explore the idea and understand how it is poised to shape the future of data management.

What is data governance?

Where data analysis is the practice of collecting, transforming, visualizing, and utilizing data to inform decision making, data governance involves the establishment of policies, procedures, and controls to ensure data quality, integrity, privacy, and security. It’s about defining who has access to what data, how the data should be used, and maintaining compliance with regulatory requirements like GDPR, HIPAA, or other industry-specific standards that apply to your organization.

Traditionally, data governance has been a manual and resource-intensive process which is prone to human error. However, automation can streamline these processes with much more efficiency and effectiveness.

Before we move on, I would like to note that, as with all things involving automation, I do recommend including some element of regular human-led oversight to ensure everything functions as intended. If erroneous automations are left to their own devices and decide to go off the rails, whatever they control will quickly suffer from a lack of regular oversight.

Why should data governance be automated?

Keep pace with scale and complexity

As a business grows, so does their data. While it may seem fine to manage data policy in a smaller organization, this task quickly becomes unfeasible as a company expands both internally and externally; internally, with respect to access control and permissions monitoring, and externally, with respect to data volume and security.

Real-time monitoring

Automation allows you to monitor your data usage in real time, which in turn ensures more immediate responses to violations or discrepancies.

Improved consistency and efficiency

By implementing automation you are simultaneously ensuring the consistent application of your governance policies, reducing the error stemming from human-led governance.

Guaranteed compliance

Automated processes can be configured to comply with the latest data privacy regulations, making sure your compliance never lapses.

How can automation enhance data governance?

Quality monitoring and cleaning

Automations can be configured to monitor data quality continuously, so flagging and remediation of inconsistencies or bad data can be carried out even if you have a constant stream of incoming data.

Access control

Perhaps an obvious place to implement automation is with access rights and permissions. As an organization grows, it should not be granting access rights to individual users as it may have been done with a smaller team. Automations should be in place to manage access with user groups and predefined roles and rules.

Managing metadata

Using automated processes to extract and manage metadata can provide valuable insights into data lineage, usage, and relevance. For example, I implemented automated metadata management on a previous team using SharePoint Document Libraries. This helped us create the versioning, rollback, and artifact creation we needed to pass a CMMI Maturity Level 3 appraisal.

Policy enforcement

Governance policies, such as data retention policies and privacy configurations, can be automated using roles and rules as mentioned before. This allows your policies to be enforced consistently across the organization.

How to get started

1. Assess the playing field.

Conduct an in-depth analysis of your existing data to identify areas that can be automated. Common examples of areas may include those mentioned in the section above.

2. Choose your tools.

Select automation tools that align with your organization’s requirements to integrate seamlessly with your existing systems.

3. Implement and test.

Use the appropriate resources to design and implement automated data governance processes. Be sure to run a sufficient amount of testing to identify bugs and defects before putting your automations into a production environment. This will help you avoid many potential hiccups during your roll-out.

4. Train and integrate.

Train your team and communicate to the organization where appropriate on your new data governance processes. Educating your team is about how it works and who to reach out to for support will help things run smoothly after launch.

Wrapping up

Automating your data governance isn’t just a new trend in tech. It’s a necessity in today’s data-centric world. By learning how to leverage automation, you can ensure your organization maintains a high degree a data quality and compliance, paving the way for improved decision-making and sustainable growth. And in doing so, you’ll watch your data management transform into a seamless, efficient, and secure process.

Using Data to Choose a Linux Distro

Scott Knight — Fri, 07 Jul 2023 16:51:29 -0500

The Problem

I recently decided that I wanted to beef up my home lab a bit by adding a small, inexpensive machine that I can use as a sandbox for server projects and testing operating systems. When I realized that StackSocial had a Lenovo ThinkCentre M900 on sale for less than $200, I ordered one.

I also knew I wanted to try something besides Windows and MacOS on this new machine, so I decided to evaluate Linux distros. As it turns out, choosing a Linux distro is a really tall order for someone who’s only superifically familiar with the domain; there are hundreds, if not thousands, of distributions.

So, knowing the membership to be Linux and FOSS fans, I turned to the Mastodon community to help me decide what to try out first.

Data Collection

I posted a poll on Mastodon asking everyone to vote for their favorite Linux Distro.

I didn’t expect to get much engagement as a new member with few followers, but it seems as though this topic is something everyone has an opinion on.

Having set the poll to stay open for a week, I received 58 votes, not accounting for recommendations from replies on the post (I asked everyone with an “Other” vote to specify their preference in a reply).

After a week, I combed through the comments to create the full dataset analyze the results.

Analysis

I used a Jupyter Notebook for the basic analysis. Pandas helped me to import and transform the data, and I used Matplotlib for the visualizations. As there were two types of recommendations, I classified the recommendations as either distros or desktops (what’s the difference?).

From there, I added a few blocks of code to clean things up, sort the data, and create the bar charts.

Results

Now that I had a couple of visuals to evaluate, the results were quite interesting and unexpected. Zorin OS was one of my personal preferences going into the poll, but it hadn’t received a single vote (since the poll closed, I’ve had some comments recommending it).

Linux Mint was by far the winner in the distro category, followed by Fedora and NixOS. So, it looks like I’ll be starting off with Linux Mint.

For desktop environments, KDE Plasma was the most recommended.

I’ve since posted the results on Mastodon, in case anyone who participated was interested in the overall outcome.

I’ve pushed the project files to my Github, but I recommend checking it out here using the Jupyter Notebook Viewer if you just want to see the results.

Scott Knight's Blog