DQ Framework

I came across the excellent draft of the new EU data quality framework that aims to align stakeholders. While it focuses on medicines regulation and procedures that apply across the European Medicines Regulatory , it definitely applies to larger data governance policies and serves as a great template for setting your own strategy.

Data quality is one of the biggest asks from consumers of the data in large scale implementations. As the scale of the data grows, so does complexity and the plethora of usecases that are now dependent on this data. In a large organization, there are typically multiple degrees of separation i.e. siloes between producers of the data and consumers. While the technology to detect and provide feedback loops to data anomalies exist, the fundamental problem of minimizing the value of clean data at the source persists.

A downstream data engineering team stitching together various sources of data is caught between a rock and a hard place i.e. between ever ravenous consumers of data/dashboards/models versus producers who throw over application data over the fence ensuring operational needs are met while downplaying analytics needs.  This also leads to minimal understanding of the data downstream by teams building and using these data products based on resourcing as the focus is on throwing data into the data lake hoarding data. The lack of understanding  is  further compounded by an ever increasing backlog of balancing the tight wire between new data products and tech debt.

This is where organizational structures play a huge part in determining the right business outcome and usage of the data. The right org structure would enforce data products conceptualized and built at source with outcomes and KPIs planned at the outset with room for evolution. 

Some of the better implementations I have been fortunate to be involved in treated data as a product where the data consumption has evolved though deliberation of usecases with KPIs defined at the outset in terms of value of this data as opposed to ‘shovel TBs into the lake’ and then we figure out what to do with it.

Call it what you want ( data mesh etc) , but fundamentally the data-driven approach to plan data usage and governance with the right executive sponsorship holds a ton of value especially in the age of massive data generation across connected systems. The advantage of being KPI-driven at the outset means you can set taxonomy at the beginning, the taxonomy helps feed the domain-based usecase which has implications for consumption of the data and sets the foundation for a holistic permissioning matrix, least privileges clearly implemented and policy-based data access. More to come on this subject but great to see a well defined manifesto defined by policy-makers at the highest levels.

Zone of Proximal Development

The best times of productivity often happen when in a state of “flow” – say a sustained state of seamless productivity while coding, a burst of self-expression while composing music, a sublime feeling of sentient exhilaration while working out and more. However, as great as that state is, its become increasingly hard over the years to achieve for me with all the various dopamine rollercoaster of distractions and the overwhelming demands of time. I came across constructivist methods while researching active learning methods and Vygotsky is a name that is prominent in this subject.

ZPD was a theory proposed by Lev Vygotsky in the 1930s. The crux of his proposal was that giving children experiences that better supported their learning enhanced their development. The 1930s were an interesting time where there was much debate about the the best methods of education and development for children. Vygotsky introduced the concept of ZPD to criticize the psychometric-based testing in Russian schools. He argued that academic tests where students displayed the same academic level were not the best way to determine intelligence. Instead, he focused on the augmented ability of children via social interaction to solve problems via interacting with more knowledgeable persons. He also inferred that while self-directed curiosity based approached worked for some subjects like languages , their learning in other subjects like Mathematics benefited by interacting with more “knowledgeable” folks.

Essentially ZPD is the place where once cannot progress in their learning without the interaction of a person who is high skilled at that learning objective. In a group setting, if some individuals grasp the concept, while other individuals are still in ZPD, the peer interaction between may create the most conducive environment for learning. While it makes sense at first glance, the nuance here is the ability to recognize and set learning objectives that are within the range of ZPD to enhance learning as well as ensure the right subject matter experts around in order to help complete that objective. ZPD can be applicable to various situations, everything ranging from the learning patterns of my 8-year old to setting appropriate goals to someone on my team for their advancement.

More interestingly, reflecting upon my learning blocks over the years, I have noticed multiple instances when serendipitous encounters/conversations have unblocked what seemed an impasse. It may not be too much of a stretch to apply this to religious philosophies like say Buddhism where the role of a teacher important for attainment  of one’s own “inner guru” to become a guru.

ZPD and leading teams

One of the key areas for being successful leaders is goal setting for your teams. Analyzing the tasks that are in ZPD for your team with a view to their future aspirations is invaluable.

Before goal setting, its worth the time to analyze the tasks that are within ZPD.  Then validate if the team has a variety of skill sets and expertise that can help mentorship for team members that are within ZPD but need that extra expertise to help them through it. For example, if a team member is tasked with an important large scale organizational objective, pair them with a partner that is more experienced to help them accomplish that task.

Delegating your own tasks to folks who are interested in being managers and being the  subject matter expert that helps them navigate ZPD. Challenging oneself to think about the types of learning we might or might not be doing on a daily basis and the effect of ZPD can help enhance that learning process.

Contemplating on my own learning, as it currently stands, is a daily dance around various concepts based on time, interest and compulsions – the patchwork quilt of topics consumed (tech, news etc) could use a recipe with ZPD in mind on topics that need to be reenforced or are important to enhance.

Interesting reads:

Learning Computer Science in the “Comfort Zone of Proximal Development”

Great paper on Vygotsky’s Zone of Proximal Development: Instructional Implications and Teachers’ Professional Development