Category Archives: Economics

How the Azure Analytics (R)evolution will Alter the Hard Truth about Data Science

If you are in technical sales or trying to break into Big Data or Data Science this post is definitely a must-read for you. It is based on real-life examples, research and analysis from a practitioner stand-point..

There is a sad but realistic truth about Data Science (DS henceforth) and Big Data (BD). There is an unfortunate dichotomy in this space:  Those who are on the inside track (the Insiders), and those who are trying to get in (theOutsiders).  In the world of the “Insiders” there exists a persistent message which garnered support amongst employers which amounts to touting the Outsiders and raising the entry barrier. How? By raising the number of skills required. I came up with a list of must-have skills for a BD or a DS role based on my own analytics exercise on hundreds of DS/BD job descriptions I gathered from a web scraping. So guess how many skills are required? Twenty two! Setting the bar at least at twenty two skills, and counting, is ridiculously unrealistic!

 Enter Microsoft!!

The will-be DS/BD game changer happened quietly with not much coverage in the tech media world. This was Microsoft’s acquisition of Revolution Analytics. So, how does the Microsoft acquisition have to do with Outsiders wanting to get into Data Science? A Lot!! Let me explain.

The one fundamental skill which is a must-have in DS is Statistics! And what does the DS community use as a Statistics programming language? R!.Here is where the story gets interesting.

A branch of DS is Machine Learning or Predictive Analysis. This is the true value-add for any BD initiative: to be able to make a data-driven decision about future strategy.

To do this in R, the very popular Analytics language, you need to go through training, etc, become an expert before you can attempt to get close to solving problems. R is archaic. It is a reincarnation of languages called S and S-Plus and it is 40-years old.

The great news is that Microsoft just acquired Revolution Analytics (hence the post title). Revolution R (a product of Revolution Analytics) is known for enabling parallel processing of R resulting in massive performance gains.

Today, R is supported in Azure Machine Learning in its current format. So why does this matter? The reason is very simple. There is talk that Microsoft will probably take R, modernize it and make it a first class language, add it to Visual Studio with Intellisense support and allow everyone to develop Azure ML solution in Visual Studio ready to  be published directly to Azure. The net effect:

  •  It will be nothing short of a coup for the Outsiders in the DS/BD field who can now develop solutions without the need for the massive R learning curve cost.
  • Azure ML already provides a lot of the R functionality today out-of-the-box so it’s a natural extension to the existing functionality
  • Azure ML makes it very simple to share code, models, and common problem solutions thus:
  • I will get my solutions at 10X time-to-market
  • I can profit from my work if I choose to publish it in the Azure Gallery
  • It will change the market dynamics as it will increase the short supply of talent of BD/DS
  • And will unleash the genius of statisticians, business users with minimal programming experience to conduct their own experiments.
  • Coupled with HDInsight which has access to Analytics APIs, this is the more efficient BD solution on the market today


What’s more: Keep an eye out on the space. As I mentioned, if you are in Microsoft technical sales, this will be your ticket to moving large enterprises onto Azure with a strategy rather than just a plain tactical need!! If you are evaluating DS technology today, you probably know very little about Azure ML and the functionality provided. I would highly encourage to evaluate Azure ML before heading to a Cloudera or MapR or any other Hadoop vendors. You will not be disappointed!


About Me:

I am a freelance consultant with over 24 years of experience in IT, strategy and Economics. I specialize in Cloud, Data Architecture and DS, Machine Learning, corporate strategy and provide architectural consulting, training, technical research, data-driven decision making solutions based on economics-based statistical methods grounded in scientific frameworks.

The 22 Skills List of Data Science:

  1. R Programming
  2. Getting and Cleaning Data
  3. Exploratory Data Analysis
  4. Reproducible Research
  5. Statistical Inference
  6. Regression Models
  7. Practical Machine Learning
  8. Developing Data Products
  9. Data Visualization
  10. DBA
  11. Hadoop (including Azure HD Insight technology stack
  12. Orchestrate data workflows
  13. Data ingestion/curation using Pig, Hive, Sqoop or other Hadoop tools
  14. Hadoop cluster configuration using Hadoop big data architecture
  15. High-level design using Business Analysis, Microsoft Azure Platform Knowledge, Blob Storage API Knowledge
  16. Blob Storage API Knowledge
  17. Metadata management tool.
  18. Model client data
  19. Mapping
  20. Data profiling – Information analyzer/Excel preferred
  21. Decide how data is going to be used to make decisions, and
  22. Knowledge of both tools and methods from statistics, machine learning, software engineering, as well as being human and show persistence

How Applied Software Economics Can Solve Technology Providers Problems

In an earlier post, I talked about the role of Economics in the Software Industry, or lack thereof, and how I embarked on a journey to seek a Master’s Degree in Business Economics to seek out the truth of whether or not economics can be applied to software technology to solve some of the problems ailing the industry.

I pursued my economics degree in earnest to investigate whether or not there are potential benefits/explanations in applying an interdisciplinary approach and if the application of certain economic concepts could have the potential to positively impact the software industry. The software industry which is riddled with many infamous stories of epic failures (!!), monopolistic behavior, collusion (see this amazing post on a secret non-poaching pact in Silicon Valley titans of technology),

All of the companies I worked for did not employ anyone with an Economics Degree (even if they did, that was not their primary job), just simply Finance! Almost the entire middle management layer are financial planners with a massive preoccupation of the 30/60/90 day budget/revenue planning/forecast, revenue attainment, etc, cycle which occurs, four times a year and they were busy making calls to folks in the field asking whether they will close the deal or not! You know the type and you almost always feel that while you are doing actual valuable work, they are just counting the beans you are bringing in!!

Anyway, to offset this massive resource drain of corporate resources of “Finance” layers, groups started sprouting up within the boundaries of the corporation and sometimes through a recommendation from an outside firm (a Management Consulting firm) with the groups’ primary focus is on Corporate Strategy, or mid to long-term planning, or sometimes the highly misnomer R&D!!. The cost of not doing so proved fatal to many companies who failed to “plan” to compete, or simply ignored to have a compete strategy as they were blindsided by smaller more nimble technology startups which overtook them. I bet you can name ten of those tech companies right now that no longer exist. You know, the software darlings of the times with the meteoric rise and speed-of-light fall.

There has been many books, research papers, etc, which study the failure of businesses. I had to read through quite a few in my recent University days (in Economics it’s called “Creative destruction” and many other terms related to Darwin’s theory of evolution). As an example, Apple and Microsoft are a great case study. At some point Apple’s “closed” business model almost cost it its own existence. However, when the PC world became the Wide Wild West of cheap components, buggy software, and so on, Apple’s business model forged ahead with a simple advantage “more stability and security”.

The economic principles behind both camps, Apple and Microsoft, were at the opposite end of the spectrum. I am talking hardware-wise. Apple was “locked, proprietary” and a “closed” ecosystem where MSFT was “Open”.

Enough theory and back to reality. If you are a business owner, executive, IT Manager, I hate to break the bad news to you: There is a cost to doing nothing! Please allow me to explain: You built a product or a service, now you are ready to sell it, how do you price it? Do you bundle it with another product? Do you follow the herd of free then premium? Build it, let will come, and then figure out a successful revenue model?! How about lock-in? Can you assure you customers they are not going to be locked in for the rest of the life of your software? The list is very long and I have yet to scratch the surface and what’s more scary is going at it alone without the aid of any theory which could be applied to address such issues.

For example of applied economics to solved project failure is an approach to treat software functionality completion as stock options where the maturity date is the completion date for that particular functionality (I borrowed this one from Barry Boehm). This is where a completion of say a report accounts for X number of options pre-agreed upon prior to the project start date. Missed the date: Your options are underwater and it’s time to focus on not missing the next cycle.

Ideas such as this one, grounded in economic theory can greatly improve performance, motivate people, and are extremely creative to solve chronic problems (how many times have you heard about projects missing deadlines with cost overruns, etc.)…. Just food for thought!

IT $pending: Does it affect your company’s size?

There is an age-old question which should be on the minds of tech folks and tech sales folks alike: What happens when firms spend money on Software Technology? It’s a deceptively simple question, however, when you dig deeper you find it one of the toughest questions to answer.

A little bit of history is in order here: The question was originally posed when an economist (Ronald Coase) questioned the wisdom of treating the Firm as a blackbox. Blackbox as where Input goes in one end, and output comes out the other with economists never paying attention to what happens inside the box. The answer was also part of my graduating Thesis at the University of Strathclyde, Department of Economics.

It’s a very interesting and a very important question especially if you are in the technology business. Let’s say you want to invest in a multi-million dollar Business Process Management System, and you CFO/CEO asks you, is this going to reduce my headcount and cut labor cost? This is a very similar question to what would happen if a Firm were to outsource a specific function. Does the outsourcing result in reduction of headcount, or expansion in business activities? Pretty tricky question! And you need to have the answers handy before you pitch any proposal to the savvy CxO level person. Remember there are people’s livelihoods hanging in the balance including the IT department you deal with on a daily basis.

Let’s zoom back out a bit here: It is well-known that the aim of the firm, and by extension, IT spending in it, have a shared goal in common, lowering the friction and costs which might arise from such friction within the firm. Therefore, if the IT project is successful (Big IF), then naturally, the friction costs should go down meaning the freeing up of resources and the increase of organization surplus and the freedom to carry out business expansionary activities if all the activities are carried within the firm and not utilizing an assembly line of contractors to do the job each has been designated to do.

So what does research tells us: According to a paper entitled: “An Empirical Analysis of the Relationship Between Information Technology and Firm Size” by one of my favorite Technology Economist of MIT, Erik Brynjolfsson, et al (link here), the evidence and answer is highlight dependent on the organization itself.

Let me clarify: If there are a lot of low-level jobs in the firms to be automated away by technology then those jobs are gone forever. You might say that upskilling those folks to perform higher-level jobs is a possibility, however, it is highly unlikely. This is short team. Long-term however, if there are plans in place to expand the business and there is a need for domain-knowledge, those folks who were automated away in the short-term, can be repurposed and gains can be reaped based on the strategy the company has in place.

From an employee stand-point though, skilling up should be a never-ending endeavor to keep one’s skillset up-to-update in an ever shifting market place.  Remember, outsourcing is not all that cracked up to be and there are many ways to circumvent the outsourcing hammer.

In the next segment, I will share how outsourcing has actually fallen flat on its face and what people did to get back in the employment game through the same companies their jobs were outsourced to….

The Currency of Knowledge: How Realizing the Economic Value of your Knowledge Will Help You Succeed in Technology and Beyond

Remember the wheel? The first Plow? The domestication of animals? The first tribe who settled down on the Nile river delta or Mesopotamia and turned into communities farming the land and actually having the first surplus as archaeologists have discovered?

What happened afterwards? Ever since those social changes took place and the first signs of civilization begun to spring up, the only thing that has been changing since then is, yes you guessed it, Technology!

Everything we have achieved as a civilization since the day of the first surplus and amassing enough food supplies to survive droughts or bad weather gave way to a new breed of humans specializing in functions such as accounting, Trade, governing, minting money, inventing numbers then alphabets and it has not stopped since then.

Fast forward a few centuries and we get the printing press which made the diffusion of ideas an order of magnitude more when sharing a new invention which has been proven pivotal to the survival of our species and spreads a lot faster than previously.

Think about it: Everything you and I do today in any field around us is technology-related or driven by technology. 
In a past reality, economy drove knowledge, however, in today’s world it more of knowledge driving the economy.

Think about your job and ask yourself a question: Do you think that the majority of us today are information/knowledge brokers? Think about the explosion of inventions and meteoric growth of information created and is available today? Do you believe that those two trends necessitate that there are professionals like you and me who specialize in specific areas where we hold an over-average or an extraordinary amount of technical information/knowledge in the areas we specialize in regardless of the vertical industry itself.

What’s the point you may ask: The point is knowing that at the core, we are information/knowledge brokers offering our clients advice on what to do, but more importantly, what not to do, and what they need versus what they want based on the tremendous amount of knowledge we possess which is our key to reaching any goal or objective we have set for ourselves.­ It’s the knowledge we posses is what holds the key to our, and our clients’, success.

Did I lose you? Ok, let me restate this: the foundation of everyone’s career is based on his/her knowledge in the area they specialize in. Achieving awareness of this fact with some introspection on what you do day-in, day-out, is guaranteed to have an impact on how you view yourself and the way you deal with your customers and colleagues. It certainly impacted how I deal with my clients (I consider colleagues as internal clients and treat them the same way I do external clients).

In one of my favorite books, Origin of Wealth: Evolution, Complexity, and the Radical Remaking of Economics by Eric D. Beinhocker, the book description states: “Accounting for the creation of wealth has long challenged humanity’s best minds” which could not be further from the truth.

The book establishes a key idea/hypothesis summarized as the ingredients to creating economic value are your usual economics foundational ideas: Land, Capital, Labor and now, Technology, as an endogenous growth factor and not an exogenous one as had been the widely held view in classical economics. Rephrasing from page 42 in the book: “The hypothesis which came to be in the mid 80’s by a Stanford economist Paul Romer…. who was became increasingly dissatisfied with the idea that that the real driver of growth, technology, was exogenous. So in 1990, Romer published a paper that kicked off the development of what has come to be known as endogenous growth theory where Romer located the source of energy for growth, not in the heroism of the entrepreneur, but in the nature of technology itself. He noted that technology has a cumulative, accelerating quality to it.” [Emphasis is mine] Simply stated: The more stuff we know, the greater the base of existing human knowledge, and the greater the payoff from the next discovery.”

This last statement is a belief which I hold very strongly and believe that the more knowledge we pass on to others, the more we are freed up to learn more.

I really like welcome your comments/opinions on this topic as I am very passionate about the subject and the very fact that technology (software) is nothing but knowledge packaged in several formats to solve business problems, empower people.