One of my older tracks (trained on a LSTM) to generate a 80s shred inspired guitar solo ( mostly done for laughs and quite embarrassing in hindsight 🙂 ) got featured on the Hi Phi Nation podcast. You can listen to the episode here:
Great episode exploring AI and machine-generated music technologies and AI-generated compositions. My AI-generated solo ( around the 45 minute mark) faced off against two guitarists which was fun and also hilarious with comments from the guitarists on the AI solo such as “reminded me of like droids from Star Wars”, “No structure to it” etc).
Generating AI-driven creativity is an extremely intriguing topic and my personal research continues on embedding AI models in chord or note sequences to enhance creativity. I think the peak experience of personalized music composition using AI has to be a human/machine collaboration (atleast until singularity occurs and blows our collective understanding of reality away). This is an exciting space with my latest muse being the open source libraries being released such as audiocraft that are powered by dynamic music generation language model. More experiments incoming.
PS: I had a whole blog post on the mechanics of generating the solo here.
With ‘The Last of Us‘ becoming everyone’s favorite live-action adaptation of a video game franchise, I fondly remember playing the game years ago along with related apocalyptic/zombie favorites like Resident Evil, Left4Dead etc. Trying to survive the hordes of the undead and boss fights after a long day of work was a trip back in the day ( and mostly less stressful than dealing with race conditions and deadlocks!) . Thanks to a full schedule (aka life), my gaming schedule has taken a backseat over the years (though with the arrival of the Steamdeck, this has seen an uptick recently). 28 Days Later rebooted the genre in the big screen and set the template for things to follow. TV Series like the Walking Dead (which I abandoned after 2 seasons once it became apparent that the story wasn’t really going anywhere), Black Summer, Kingdom etc had some good moments. However, post-apocalyptic fiction writing got me interested and playing those games and watching those series/movies in the first place. The genre is rich with amazing books and a lot of mediocre ones as well. The gloomy Seattle weather plays a perfect conduit to huddling up with desolate dystopian stories. Some of my favorites listed below and could be considered essential reading for this genre.
World War Z by Max Brooks – The “oral” history about an apocalyptic war with the undead that almost wiped out the human race as we know it. This reads almost like your favorite travel vlogger visiting various sites decimated by the undead. The book actually offers an interesting commentary of how governments could (mis)manage an event of catastrophic proportions while the world is amidst a meltdown. The movie was nothing great frankly and the book offered a much more interesting and sometimes humorous view of how a society could adapt against the zombie apocalypse.
The Road by Cormac McCarthy – One of the most hopeless, desolate, heart-breaking stories in the genre. I’ve read this one twice and its one of those books where the writing is alive and makes you plod till the end with the wretched protagonists despite the utter futility you already know that awaits in the end. A father and son trudging through a devastated wasteland trying to survive against a world that includes diseases, starvation, cannibals and more. Its a road to nowhere with only your humanity to treasure. The movie was one of the rare well-made book-to-TV adaptions and remained sincere to the book.
“Borrowed time and borrowed world and borrowed eyes with which to sorrow it.” -Cormac McCarthy, The Road
The Stand by Stephen King – Not part of zombie genre but the 1000 page uncut version of the ultimate post-apocalyptic storyline by master of the macabre is essentially reading for this genre. The battle of good versus evil against a background of disease and complex human relationship finally ending with a showdown in Vegas. Nothing not to enjoy on this one. As a huge king fan, I’ve enjoyed most of his books during my teenage and adult life and this one left a big impression on me. This one is due a re-read.
I am legend – Richard Matheson – Post-apocalyptic vampire-like zombies against the last man on earth who locks his doors and prays for dawn every night. This was one of those books where the second read was more enjoyable than the first. The main characters search for purpose and the utter isolation is one of the more enjoyable aspects of this book. Its a lot different than the Will Smith movie where he slaps the zombies around and goes out in a blaze of glory ( in one of the endings at least).
The Dog Stars by Peter Heller – No zombies on this one but the narrator ( a pilot) and his friend protect their abandoned airport from human scavengers in a post-pandemic situation. Things pick up when the main character has enough and starts to search out for civilization. This one deals with friendship, loneliness, trust and the insignificance of mankind against the power of nature.
I came across the excellent draft of the new EU data quality framework that aims to align stakeholders. While it focuses on medicines regulation and procedures that apply across the European Medicines Regulatory , it definitely applies to larger data governance policies and serves as a great template for setting your own strategy.
Data quality is one of the biggest asks from consumers of the data in large scale implementations. As the scale of the data grows, so does complexity and the plethora of usecases that are now dependent on this data. In a large organization, there are typically multiple degrees of separation i.e. siloes between producers of the data and consumers. While the technology to detect and provide feedback loops to data anomalies exist, the fundamental problem of minimizing the value of clean data at the source persists.
A downstream data engineering team stitching together various sources of data is caught between a rock and a hard place i.e. between ever ravenous consumers of data/dashboards/models versus producers who throw over application data over the fence ensuring operational needs are met while downplaying analytics needs. This also leads to minimal understanding of the data downstream by teams building and using these data products based on resourcing as the focus is on throwing data into the data lake hoarding data. The lack of understanding is further compounded by an ever increasing backlog of balancing the tight wire between new data products and tech debt.
This is where organizational structures play a huge part in determining the right business outcome and usage of the data. The right org structure would enforce data products conceptualized and built at source with outcomes and KPIs planned at the outset with room for evolution.
Some of the better implementations I have been fortunate to be involved in treated data as a product where the data consumption has evolved though deliberation of usecases with KPIs defined at the outset in terms of value of this data as opposed to ‘shovel TBs into the lake’ and then we figure out what to do with it.
Call it what you want ( data mesh etc) , but fundamentally the data-driven approach to plan data usage and governance with the right executive sponsorship holds a ton of value especially in the age of massive data generation across connected systems. The advantage of being KPI-driven at the outset means you can set taxonomy at the beginning, the taxonomy helps feed the domain-based usecase which has implications for consumption of the data and sets the foundation for a holistic permissioning matrix, least privileges clearly implemented and policy-based data access. More to come on this subject but great to see a well defined manifesto defined by policy-makers at the highest levels.
Most times when I have taken over new roles or added more portfolios to my current workstreams, it has usually involved decision making to either build on or completely overhaul legacy code. I’ve truly encountered some horrific code bases that are no longer understandable, secure or scaleable. Often if there is no ownership of the code or if attrition has rendered it an orphan, it usually sits unnoticed ( like an argiope spider waiting for its prey) until it breaks (usually on a friday afternoon) when someone has to take ownership of it and bid their weekend plans goodbye. Rarely do teams have a singular coding style with patterns and practices clearly defined that are repeatable and can withstand change people or technology change – if you are in such an utopian situation, consider yourself truly lucky!
A lot of times, while you have to understand and reengineer the existing codebase, it is imperative to keep the lights on while you are figuring out the right approach, a sorta situation while you got to keep the car moving while changing its tires. I discovered Tenacity while encountering a bunch of gobbledygook shell and python scripts crunched together that ran a critical data pipeline and had to figure out a quick retry logic to keep the job running while it randomly failed due to environment issues ( exceptions, network timeouts, missing files, low memory and every other entropy-inducing situation on the platform). The library can handle different scenarios based on your usecase.
Continuous retrying for your commands. This usually works if its a trivial call where you have minimal concerns over overwhelming target systems. Here, the function can be retried until no exception is returned.
Stop after a certain number of tries. Example – if you have a shell script like shellex below that needs to execute and you anticipate delays or issues with the target URL or if a command you run raises an exception, you can have Tenacity retry based on your requirement .
Exponential backoff patterns – To handle cases where the delays maybe too long or too short for a fixed wait time, tenacity has a wait_exponential algo that can ensure that if a request cant succeed in a short time after a retry, the application can wait longer and longer with each retry thus alleviating the target system of repetitive fixed time retries.
The library handles plenty of others uses cases like Error handling, custom callbacks and tons more.
Overall, this has been a great find to use a functional library like Tenacity to for various usecases instead of writing custom retry logic or implementing a new orchestration engine for handling retries.
I primarily wanted to read this book due to the Pytorch section and pretty much flipped through the Scikit learn section while absorbing and practicing with the Pytorch section. So my review largely is based on chapter 13 and beyond. Apart from the Pytorch official documentation, there are not too many comprehensive sources that can serve as a good reference with practical examples for Pytorch in my view. This book aims to do that and pretty much hits the mark.
Ch 1-11 is a good refresher on SciKit -learn and sets up all the foundational knowledge you need for the more advanced concepts with Pytorch. I wish though the authors had different examples and not resorted to ubiquitous examples like MNIST ( like in chapter 11/ 13 etc) for explaining neural network concepts. While these are good for understanding foundational concepts, I find the really good books usually veer away from standard examples found online and get creative.
Chapter 12 provides an excellent foundation in Pytorch and a primer for building a NN model in Pytorch. The code examples are precise with the data sources clearly defined so I could follow along without any issues. I did not need a GPU/collab to run examples . Good to see the section on writing custom loss functions as those are useful practical skills to have.
Ch-14 which has us training a smile classifier to explain convolution neural networks is a useful example especially for tricks like data augmentation that can be applied to other usecases.
I skipped through the chapter on RNNs as transformers are the rage now ( Ch-16) and Pytorch already has everything implemented in its wrapper function for structures like LSTMs. Still , a lot of dense and useful material explaining the core concepts behind RNNs and some interesting text generation models using Categorical pytorch classes to draw random samples.
The chapter on Transformers is a must-read and will clear up a lot of foundational concepts. Another thing to mention is that the book has well depicted color figures that make some of the dense material more understandable. Contrasting the transformers approach to RNNs using concepts like attention mechanisms is clearly explained. More interestingly, the book dwells into building larger language models with unlabeled data such as BERT and BART. I plan to re-read this chapter to enhance my understanding of transformers and the modern libraries such as HuggingFace that they power.
The chapter of GANs was laborious with more MNIST examples and could have had a lot more practical examples.
Ch-18 on Graph Neural Network is a standout section in the book and provides useful code examples to build pytorch graphs to treat as datasets defining edges and nodes. For example, libraries like Torch Drug are mentioned that use pytorch Geometric framework for drug discovery. Spectral graph convolution layers, graph pooling layers, and normalization layers for graphs are explained and I found this chapter to be a comprehensive summary that would save one hours of searching online for the fundamental concepts. GNNs definitely have a ton of interesting applications and a lot of recent papers with links are provided.
Ch-19 on reinforcement learning adds another dimension to the book which is largely focused on supervised and unsupervised learning in prior chapters. Dynamic programming to Monte Carlo to Temporal Difference methods are clearly articulated. The standard open AI gym examples are prescribed for implementing grids to specify actions and rewards. I thought this chapter was great explaining the theoretical concepts but the examples were all the standard Q-learning fare you would find online. Would have loved to see a more realistic example or pointers to apply to your own usecases.
All in all, I enjoyed the Pytorch examples and clearly explained concepts in this book and it would be a good Pytorch reference to add to your library.
The usual Synchronous versus Asynchronous versus Concurrent versus Parallel is a topic in technical interviews that usually leads to expanded conversations in the candidates overall competency on scaling and leads to interesting rabbit holes and examples. While keeping the conversations open-ended, I’ve noticed when candidates usually incorporate techniques to speed up parallelism or enhance concurrency or mention other ways of speeding up processing, its a great sign.
Its also important to distinguish between CPU-bound and IO-bound tasks in such situations since parallelism is effective on CPU-bound tasks (example preprocessing data, running ensemble of models) while concurrency works best for IO-bound tasks (web scraping, database calls). CPU-bound tasks are great for parallelization using multiple CPUs where a task can be split into multiple subtasks while IO-bound are not CPU dependent but depend on time reading/writing from disk.
Key standard libraries in Python for concurrency include:
AsyncIO– for concurrency with coroutines and eventloops. This is most similar to a pub-sub model.
Both are limited by the global interpreter lock (GIL) and single process, multi-threaded. Note for parallelism, the library I’ve usually used is multiprocessing which is another post.
AsyncIO is a great tool to execute tasks concurrently and is a great way to add asynchronous calls to your program. However, the right usecase matters as it can also lead to unpredictability since tasks can start, run and complete in overlapping times using context switching between threads. Threads can be blocked using asyncio and the next available thread in the queue can be processed until it completes or is blocked. The key here is the lack of any manual checking if the thread is freed up, as asyncio will announce the availability of the thread when it does actually free up.
This post has a couple of quick examples of asyncio using the async/await syntax for event-loop management. Plenty of other libraries available but async is usually sufficient for most workloads and a couple of simple examples go a long way in explaining the concept.
The key calls here are –
async – tells the Python interpreter to run the coroutine asynchronously with an event loop
await – while waiting for the result to be returned, this passes control back to the event loop, suspending the execution of the current coroutine to let the event loop run other things until the await has a result returned
run – Schedule a coroutine (python 3.7+) . In earlier versions (3.5/3.6) you can a
sleep – suspends execution of a task to switch to the other task
gather – Execute multiple coroutines that need to finish before resuming the current context with the list of responses from each coroutine
Example of using async for multiple tasks:
A trivial example like above goes a long way in explaining the core usage of the sync library especially while bolting it onto long running python processes that are primarily slow due to IO.
aiohttp is another great resource for synchronous HTTP requests which by nature are a great usecase for asynchronicity while requests wait for servers to respond to do other tasks. This basically works by creating a client session that can be used to support multiple individual requests and make connections upto 100 different servers at the same time.
Non async example
A quick example to handle requests from a website (https://api.covid19api.com/total/country/{country}/status/confirmed) that provides a JSON string based on the specific request. The specific request is not important here and used only for demonstrative purposes to demonstrate the async functionality.
Async example using aiohttp which will be needed in order to asynchronously call the same endpoint.
The example clearly shows the time difference where the async calls halve the time taken for the same calls. Granted its a trivial example but it shows the benefit of the non-blocking Ascync call and can be applied to any situation that deals with multiple requests and calls to different servers.
Key Points
AsyncIO is usually a great fit for IP bound problems
Putting async before every function will not be beneficial as the blocking calls can slow down the code so validate the usecase first
async await support a specific set of methods only so for specific calls (say to databases), the specific python wrapper library you use will need to support async await.
I’ve enjoyed the different eras of Genesis regardless of the Peter Gabriel-era prog-rock or the later more commercial Collin-era hits. Regardless of the timeline, the Hackett-Rutherford guitar soundscapes made me love the catalog whether it was the theatrics of Nursery Cryme, the sonic bliss of Selling England by the Pound, the massive hooks of Invisible Touch, or the 90s heyday of We cant Dance ( a personal guilty pleasure).
On the spur of the moment, a couple of high school buddies and I decided to fly to NYC to see the Genesis – ‘The Last Domino’ tour at Madison Square Garden. Being progheads, the motivating factor was to catch Genesis on what could be their final tour . Being someone who spent his teens in the 90s in a cloud of personal angst, the guitar was an escape and I have many fond memories of spending many a gray bleak day muddling my way through a Genesis guitar tab to manifest some poorly played yet soul satisfying covers. With so many of my musical heroes passing over the last few years ( Neil Peart, Eddie Van Halen, Allan Holdsworth…this list goes on sadly), its an imperative now to see the acts I have never had the chance to see before or want to see again.
The Last Domino
The setlist was average and largely meant to accommodate Phil Collins’ current state of performance prowess. I had no expectations of them belting out “”Get ‘Em Out By Friday” or “Can Utility And The Coastliners'” . However, with Nic Collins being stellar on drums and Phil still belting out the songs in top form backed by the genius of Rutherford/Banks , I was glad I made the trek and it was a show to remember not merely for nostalgia but that we could actually make it out to a legit concert in 2021 after an year(s) of pandemic-induced seclusion.
The best times of productivity often happen when in a state of “flow” – say a sustained state of seamless productivity while coding, a burst of self-expression while composing music, a sublime feeling of sentient exhilaration while working out and more. However, as great as that state is, its become increasingly hard over the years to achieve for me with all the various dopamine rollercoaster of distractions and the overwhelming demands of time. I came across constructivist methods while researching active learning methods and Vygotsky is a name that is prominent in this subject.
ZPD was a theory proposed by Lev Vygotsky in the 1930s. The crux of his proposal was that giving children experiences that better supported their learning enhanced their development. The 1930s were an interesting time where there was much debate about the the best methods of education and development for children. Vygotsky introduced the concept of ZPD to criticize the psychometric-based testing in Russian schools. He argued that academic tests where students displayed the same academic level were not the best way to determine intelligence. Instead, he focused on the augmented ability of children via social interaction to solve problems via interacting with more knowledgeable persons. He also inferred that while self-directed curiosity based approached worked for some subjects like languages , their learning in other subjects like Mathematics benefited by interacting with more “knowledgeable” folks.
Essentially ZPD is the place where once cannot progress in their learning without the interaction of a person who is high skilled at that learning objective. In a group setting, if some individuals grasp the concept, while other individuals are still in ZPD, the peer interaction between may create the most conducive environment for learning. While it makes sense at first glance, the nuance here is the ability to recognize and set learning objectives that are within the range of ZPD to enhance learning as well as ensure the right subject matter experts around in order to help complete that objective. ZPD can be applicable to various situations, everything ranging from the learning patterns of my 8-year old to setting appropriate goals to someone on my team for their advancement.
More interestingly, reflecting upon my learning blocks over the years, I have noticed multiple instances when serendipitous encounters/conversations have unblocked what seemed an impasse. It may not be too much of a stretch to apply this to religious philosophies like say Buddhism where the role of a teacher important for attainment of one’s own “inner guru” to become a guru.
ZPD and leading teams
One of the key areas for being successful leaders is goal setting for your teams. Analyzing the tasks that are in ZPD for your team with a view to their future aspirations is invaluable.
Before goal setting, its worth the time to analyze the tasks that are within ZPD. Then validate if the team has a variety of skill sets and expertise that can help mentorship for team members that are within ZPD but need that extra expertise to help them through it. For example, if a team member is tasked with an important large scale organizational objective, pair them with a partner that is more experienced to help them accomplish that task.
Delegating your own tasks to folks who are interested in being managers and being the subject matter expert that helps them navigate ZPD. Challenging oneself to think about the types of learning we might or might not be doing on a daily basis and the effect of ZPD can help enhance that learning process.
Contemplating on my own learning, as it currently stands, is a daily dance around various concepts based on time, interest and compulsions – the patchwork quilt of topics consumed (tech, news etc) could use a recipe with ZPD in mind on topics that need to be reenforced or are important to enhance.
Merkle trees (named after Ralph Merkle, one of the fathers of modern cryptography) are fascinating data structures used in hash based data structures to verify the integrity of data in peer-to-peer systems. Systems like Dynamo use this to compare hashes – essentially itself a binary tree of hashes and typically used to remove conflicts for reads. For example – in a distributed system, if a replica node falls considerably behind its peers, using techniques like vector clocks might take unacceptable times to resolve. A hash-based comparison approach like Merkle tree would help quickly compare two copies of a range of data on different replicas. This is also a core part of blockchains like Ethereum which uses a non-binary variant but the binary ones are the most common and easy to understand and fun to implement.
Conceptually this involves:
Comparing the root hashes of both trees.
Continue recursion on the left and right children of the tree until the root hashes are equal.
The “Merkle root” stores the summary of all the transaction value in a singular value.
Simple Example
For example , if TA, TB,TC ,TD are transactions ( could be files, keys etc) and H is a Hash function. You can construct a tree by taking the transactions, hashing their concatenated values to generate children and finally reduced to a single root. In my scrawl above, this means hashing TA and TB, TC and TD, then hashing their concatenations H (AB), H(CD) to land at H(ABCD).Essentially keep hashing the until all the transactions meet at a single hash.
Example
Here’s an example that uses this technique to compare two files by generating their Merkle root to validate if they are equal of not (comments inline).
Invoke the script by calling “python merkle_sample.py “<file1>.csv” “<file2>.csv” to compare two merkle trees. Code below:
Key advantage here is that each branch of the tree can be checked independently without downloading the whole dataset to compare.
This translates to reducing the number of disk reads for synchronization though that efficiency needs to be balanced against the recalculation of the entire tree when nodes leave or go down. This is fundamental to Crypto currencies when transactions need to be validated by nodes and there is enormous time and space cost to validate every transaction which can be mitigated by Merkle trees in logarithmic time instead of linear time. The Merkle root get put into the block header that gets hashed in the process of mining and comparisons are made via the Merkle root rather than submitting all the transactions over the network. Ethereum uses a more complex variant of the Merkle, namely the Merkle Patricia tree.
The applications of this range beyond blockchains to Torrents, Git, Certificates and more.
