AI Landscape after Build 2017

AI Landscape after Build 2017

Initial Thought

Before I dive into my experience of AI at Build 2017, I’d like to put things into perspective.

I came to the US from Budapest in 1989 as a tourist, 10 days after I got my masters in Electrical Engineering, determined to become a great computer programmer. After undergoing the usual immigrant hardship, I rented a house, subletting the rooms and buying substantially discounted food at 2 am at PathMark, the local supermarket, to stay afloat.
Fortunately, I had a huge finished basement (a true man cave) where I immersed myself into studying all the Windows publications available at the time: Programming Windows by Charles Petzold.
In 1991 I interviewed Jeff Street, who later became a co-worker, tenant, and friend. Together we worked at D&B. In our spare time, we experimented with the uncharted territories of Windows programming.
I can recall the wave of excitement we felt when we were able to send a wave file across our rooms. A HUGE deal!
It has been very fascinating to see the digital transformation taking place in my lifetime, PC -> DOS -> GUI -> Web -> Mobile -> AI.


[02:45:06] In human history groundbreaking inventions lead to paradigm shifts and revolution.
Pivotal moments of innovation have produced new tools and empowered bold thinkers to propel humanity forward. Powerful technology has arrived that inspires new breakthroughs. Microsoft Al is here to amplify human ingenuity. Helping others to see the world, aiding in cancer research, increasing productivity, and efficiency.

I have the same tingling feeling today about AI as I did back in 1991, full of excitements and anticipation.

Build 2017 had the same excitement about AI. In fact, AI dominated the 3-day event, followed closely by related topics: Azure IoT Edge, Bot Framework, Azure Data Lake + USQL, SQL Server 2017, VS 2017 R and Python Development tools.

We have moved from a Mobile first to an AI first era. Let’s look at the various AI topics at Build 2017.

Cognitive Services

We can measure the importance of a topic by where and how it’s presented.
2 years ago Cognitive Services was presented in a small breakout room with 4 APIs.
1 year ago in a big breakout room with 22 APIs.
This year it was part of the Key Note on the main stage with 29 APIs.

3 big forces are converging

3 big forces are converging in the industry

  1. Big compute: the advantages of huge computing power in the cloud.
  2. Powerful algorithms, with amazing progress in deep learning and the deep reinforcement learning.
  3. Massive amounts of data that fuel powerful AI applications.

Let’s take a look at how Computer Vision and Speech are driving Cognitive Services

Computer Vision

You have all heard about the deep neural networks making great progress for image recognition and for object segmentation. Two years ago Microsoft Research created a new approach to recognize images with a very deep neural network architecture called the ResNet [4]. With 152 layers it won the first place in all the three categories of the ImageNet competition. A lot of AI Developers, Data Scientists, and Core Machine Learning Engineers use ResNet today to train deep neural networks for their applications.

ResNet, 152 layer Convolutional Neural Network
ResNet, 152 layer Convolutional Neural Network


[2:17:22] Six months ago Microsoft Research had a breakthrough in speech recognition [5].
The first time AI surpassed human understanding with a stunning 5.8% error rate. Here, the famous switchboard data set was used (a collection of phone call data).

Speech Error Rate 5.8%
Speech Error Rate 5.8%

Cognitive Services provides the broadest and the deepest set of developer services for AI in the industry.
Microsoft released 5 additional services bringing the total APIs to 29 across vision, speech, language, search, and knowledge. This includes the new video indexer service. It combines a lot of features from speech to face to vision to semantic understanding.
[2:19:40] Cognitive Services, Custom Vision, LUIS, Speech, Search, and Decision Services Announcement

Cognitive Services
Cognitive Services

Microsoft opened cognitive services labs as well as new preview services. Experimenting with hand gesture recognition and listening to developer feedback provided more customization capabilities to existing services like LUIS, the Custom Language Understanding Tool.
There are deeper customizations across all Cognitive Services: Custom Vision, Custom Language Understanding, Custom Speech, Custom Search and Custom Decision services, still using REST APIs without the needs of designing very complex models. Making it so much easier to develop AI applications.

There is also a new service in Azure to train deep neural networks efficiently.
It supports all favorite deep learning frameworks: Microsoft Cognitive Toolkit or TensorFlow or Caffe or others.
Azure + AI provides a comprehensive platform with everything a developer needs to infuse AI into applications.
Learn more about Cognitive Services.

Azure + AI

Bringing AI to every developer

Azure + AI
Azure + AI
Data platformInfrastructureServicesToolsAI Frameworks
Data LakeBatch Al Training ServiceCognitive ServicesVisual StudioCognitive Toolkit
Cosmos DBFPGA, GPU, CPUBot ServiceAzure Machine LearningTensorFlow
SQL ServerBot FrameworkCaffe

[02:31:39] Check out how Shanghai Changzheng Hospital diagnoses diabetes with AI by looking at retina images.

In the home from, Windows to Xbox, from Bing to Office every Microsoft products is being transformed with AI.

AI + Home products
AI + Home products

[02:34:00] Presentation Translator add-in for PowerPoint understands and translates speech to text, a marvelous achievement.


[02:37:19] Another key product being fused with AI is Cortana.
Microsoft wants Cortana to be our personal intelligent assistant that goes wherever we go. At work, at home, at the gym, or on the go, Cortana continues to expand its reach across platforms and devices. As developers, we will now have the power to build Cortana skills using the skill kit which utilizes Microsoft Bot framework for the more than 100 million Cortana users.
Learn more at about Cortana.

Microsoft Graph

[02:38:09] Data is the magic that enables AI.
Microsoft Graph offers an unmatched data graph including work knowledge from Office 365 dynamics and from all of your devices.
As well as word knowledge from Bing and LinkedIn. By combining Microsoft Data Graph with our own business data and our understanding of our users, we are finally ready to build incredible AI applications.
Learn more at about Microsoft Graph.

Microsoft Graph + Business Data = Incredible AI apps
Microsoft Graph + Business Data = Incredible AI apps

AI is a Disruptive Technology

[02:43:45] AI can truly transform our business and disrupt every single business applications. Whether an industry is a vertical one like banking, retail, or healthcare or a horizontal business process like sales, marketing, or customer support. Microsoft wants to help us embrace AI and apply it to our business.

“The best way to predict the future is to invent it” – Alan Kay

AI disrupts vertical industries and horizontal business processes
AI disrupts vertical industries and horizontal business processes

Azure IoT Edge

[22:47] Edge computing brings cloud compute capabilities to the edge of a network, closer to IoT data, enabling to securely distribute cloud intelligence locally.
We can test and deploy using the usual cloud CI/CD pipelines, which we are already accustomed to, and push modules down to the IoT device in Docker containers. These containers are communicating via a local version of the cloud IoT Hub, like a service bus.

Someone, please pinch me. Am I dreaming or has my wildest dream come true?
So, Microsoft. Are you telling me that I can use local Machine Learning models to detect objects, people, and motion in my IoT video stream and send those up to the cloud where my super-duper custom Machine Learning models can identify objects, faces, ages, and emotions?


But wait, there is more! Azure IoT Edge runs on Windows and Linux, and on any devices with more than 128 MB memory, which is smaller than a Raspberry Pi 3.
Now really, how cool is that? Did I hear knocking? Opportunity, please make your grand entrance!

Here are the main advantages

  • Run artificial intelligence at the edge
  • Perform edge analytics
  • Deploy IoT solutions from cloud to edge
  • Manage devices centrally from the cloud
  • Operate with offline and intermittent connectivity
  • Enable real-time decisions
  • Connect new and legacy devices
  • Reduce bandwidth costs
Azure IoT Edge
Azure IoT Edge


Azure Data Lake + USQL

Azure Data Lake Store is an enterprise-wide hyper-scale repository for big data analytic workloads that enables you to capture data of any size, type, and ingestion speed in one single place for operational and exploratory analytics.
It’s built for Hadoop and is an Apache Hadoop file system, compatible with Hadoop Distributed File System (HDFS) and it makes it easy for developers, data scientists, and analysts to store data of any size, shape, and speed. It does all types of processing and analytics across platforms and languages.

Azure Data Lake
Azure Data Lake

U-SQL is the new big data language for Azure Data Lake, that unifies the benefits of SQL with the expressive power of your own code. The syntax is based on T-SQL while it uses C# types as a default.


U-SQL enables you to take advantage of Cognitive Services, like Image OCR, Vision, or Sentiment Analysis from within Azure Data Lakes in a massively parallel fashion.
This approach allows developers to develop enterprise scale solutions with AI without being a data scientist or machine learning expert.

Microsoft Cognitive Toolkit

The Cognitive Toolkit [11], aka CNTK, is a unified deep-learning toolkit, that can be accessed from Python, C, C++, .NET and Java or it can be used as a standalone machine learning tool through its own model description language (BrainScript). It supports 64-bit Linux or Windows and it is in RC3 status, to be released in June.
This is a very serious tool for a Machine Learning Engineer and similar in structure and principles to TensorFlow.
I’ll be writing extensively about Cognitive Toolkit in my future blogs. Showing you how to approach a problem, how to select the model [7]; giving you a quick overview of how the model works [6] and plenty of resources to immerse yourself in the academic details.
I highly recommend checking out the new Cognitive Toolkit course on edX starting on June 26 by Sayan Pathak, Ph.D. and Roland Fernandez.

SQL Server 2017

[9] Machine learning services in SQL Server 2017 provides Python support for in-database machine learning. We can run Python code in SQL Server and Python-based deep learning model, which is built and deployed on an SQL Server. Cognitive Toolkit and TensorFlow open source deep learning framework models wich run on an SQL Server can leverage GPU and boost model training performance. Amazing!

Key Advantages

  • Eliminate Data movement with in-database computation
  • Operationalize Python code and models
  • Achieve enterprise-grade performance and scale

Data Scientists – Data Exploration and Model Development

Working from IDE on a local workstation, execute Python code which runs in a database on a remote SQL server, and get results back.

Data Scientist
Data Scientist

Application Developer – Model Operationalization

Python Code ->T-SQL Stored Proc
Call T-SQL Stored Proc from your application to get the results from Python code execution (predictions, scores, plots etc.) in the applications

Application Developer
Application Developer

How does Python in an SQL Server work?

Python in SQL Server
Python in SQL Server

The Learning and Scoring Process

The Learning and Scoring Process
The Learning and Scoring Process

Sample Deep Learning Database Application

  • Build enterprise grade AI applications by combining the best of SQL Server & Python
  • Improve security and efficiency by eliminating unnecessary data movement
  • Accelerate application performance by tapping into the power of GPUs
Sample Deep Learning Database Application
Sample Deep Learning Database Application

Why Python?

Because it’s the most popular and fastest growing language for Data Scientists and Machine Learning Engineers [10].

Python Is the Most Popular and Fastest Growing Language for Data Science
Python Is the Most Popular and Fastest Growing Language for Data Science

Bot Framework

Last year Bot Framework had 9 channels, this year it gained 3 new channels: Cortana Skills, Skype for Business, and Bing.

Visual Studio 2017 Python Development Tools

The VS 2017 Python Development Tools is a great addition to the VS tool set. It makes it super easy and convenient to consume third-party Python packages, develop, debug, and test your deep learning models.
Visual Studio provides a comprehensive debugging experience for Python, including attaching to running processes, evaluating expressions in the Watch and Immediate windows, inspecting local variables, breakpoints, step in/out/overstatements, Set Next Statement, and more.
You can also remotely debug Python code on Azure via Websockets!

AI Learning Path

Please watch the Navigating the AI Revolution session to see the learning paths other developers took to become a AI Consumer, Data Scientist or Machine Learning Engineer.
First, become an AI Consumer by using Cognitive Services APIs.
Second, to become a Data Scientist begin by using R, Python, Azure Data Lake and/or SQL Server 2017 and have an understanding of eliminating outliers, data cleaning, feature detection, etc.
The Machine Learning Engineer path is the hardest and of course the most rewarding.
This path requires a solid understanding of deep neural networks. And while there are great templates guiding you in selecting the right model for the job [8], you must develop an intimate understanding of your problem domain, models, model parameters, and most importantly develop a great intuition to get the best results.
Data visualization and practical data manipulation are vital to getting the best data quality. Combined with feature selection it also produces fantastic results and minimal error rate.
This path requires academic knowledge with a huge dose of common sense, intuition, and of course lots and lots of practice.
I’ll write more about this in great detail in my AI Learning Path blog.


What an exciting time to learn about the new software landscape. Learn how to thrive as a developer in a world of machine learning, neural networks, and data science. Microsoft is making it easy to add AI to your applications.
Don’t be the one who is using VB while everybody around you is using operating system agnostic .NET Core in containers implementing microservices. Or the one who thinks Sitecore is the best thing since sliced bread!
This paradigm shift will be the biggest revolution in IT history and it is coming like a tsunami, sweeping everything in its path.
Please join me for a great ride at the top of the wave!

It was very sad to see when at the end of the Navigating the AI Revolution session Bill Barnes, Principal Software Engineer at Microsoft DX, said very eloquently when the crowd had a mass exodus instead of staying for an invigorating and enlightening Q&A:

“I know lunch is next. What can we compete with?”

Please value your and your families future more than your animal instincts!


  1. Microsoft AI
  2. Build 2017 – Channel 9 – AI, Machine Learning, IoT, SQL Server, R, Python
  3. Navigating the AI Revolution – Bill Barnes, Micheleen Harris
  4. Microsoft researchers win ImageNet computer vision challenge
  5. Microsoft researchers reach human parity in conversational speech recognition
  6. Data Science and Robots – Brandon Rohrer
  7. How to choose algorithms for Microsoft Azure Machine Learning – Gary Ericson, Larry Franks, Brandon Rohrer
  8. Machine learning algorithm cheat sheet for Microsoft Azure Machine Learning Studio – Gary Ericson, Larry Franks, C.J. Gronlund, Brandon Rohrer
  9. Built-in machine learning in Microsoft SQL Server 2017 with Python
  10. The Most Popular Languages for Data Science
  11. Deep learning with Microsoft Cognitive Toolkit – Sayan Pathak, Chris Basoglu
  12. Build 2017 – Keynote 1
    [00:02] Satya Nadella Opening Statements

    [07:39] Microsoft mission
    [14:02] Intelligent Cloud and Intelligent Edge
    [19:00] Sam George, Sandvik Windows IoT demonstration
    [23:00] Azure IoT Edge Introduction
    [27:52] Andrea Carl, AI for Workplace Safety demonstration
    [37:59] The Edge and our Devices
    [41:27] Laura Jones, Intelligent Meetings
    [51:46] Satya Nadella Closing Statements
    [52:50] Haiyan Zhang and Emma Lawton, Development and Inspiration
    [2:13:13] Harry Shum Opening Statements
    [2:18:18] Cognitive Services – Video Index and Cognitive Services Labs Cognitive Services Announcement
    [2:19:40] Cognitive Services, Custom Vision, LUIS, Speech, Search and Decision Services Announcement
    [2:23:00] Cornelia Carapcea Custom Vision demonstration
    [2:27:07] Bot Framework and Adaptive Cards demonstration
    [2:33:27] Yina Arenas Presentation Translator Add-in demonstration
    [2:37:44] HP and Intel Cortana Device Announcement
    [2:39:12] Yina Arenas Tact and Microsoft Graph demonstration
    [2:43:52] Harry Shum Closing Statements

Leave a Reply

Your email address will not be published. Required fields are marked *