For developers with a Microsoft .NET background who want to get familiar with building Spark applications with Scala programming language, this blog post series is a walk through from installing the development tools and building a simple Spark application, then submit against an HDInsight Spark cluster. My HDInsight configuration is Spark 2.0 (HDI 3.5) with …
Continue reading Building a Spark Application for HDInsight using IntelliJ Part 1 of 2
With Power BI Desktop you can use the Azure HDInsight Spark BI Connector to get data from the Spark cluster to build reports. I have an HDInsight Spark 2.0 cluster with Azure Data Lake Store as the primary storage. Open Power BI Desktop Click Get Data Enter url of your HDInsight Spark cluster. Enter the …
Continue reading Build Power BI Reports with HDInsight Spark Connector
Upon creating an HDInsight cluster for big data analytics development, the costs can be quite high at around $4 an hour. This adds up, especially when running idle most of the time. To save on the costs, I delete the cluster and re-create when needed. For simple development purposes, my HDInsight depends on Azure Data …
Part 1 of 2: Text Analytics of Movie Reviews using Azure Data Lake, Cognitive Services and Power BI (part 1 of 2) Take a csv file, analyze with an U-SQL script in Azure Data Lake Part 2 of 2: Applicable Business Scenario Marketing or data analysts who need to review sentiments and key phrases of a […]
Part 1 of 2: Text Analytics of Movie Reviews using Azure Data Lake, Cognitive Services and Power BI (part 1 of 2) Take a csv file, analyze with an U-SQL script in Azure Data Lake Part 2 of 2: Text Analytics of Movie Reviews using Azure Data Lake, Cognitive Services and Power BI (part 2 of 2) …
This article is for beginners looking to understand the developer experience in Visual Studio and working with hive tables in HDInsight. I developed the following HiveQL statements. My cluster is HDInsight Spark 2.0 cluster. Before executing these statements, I have the database and tables: The crimes table data looks like: Let’s query the table with …
Continue reading HiveQL Group By and Views with Visual Studio and HDInsight
Objective: From Azure Data Lake Store, copy data in a .tsv file to a database table in Azure SQL DB. My take on Azure Data Factory is that of ETL but less of the transformation and move data from on-premises and cloud sources to Azure data-oriented services. Read further about Azure Data Factory https://docs.microsoft.com/en-us/azure/data-factory/data-factory-introduction In …
Continue reading Azure Data Factory: Copy Data from Azure Data Lake Store to Azure SQL
This is an introductory walk through of querying hive tables and visualizing the data in the Ambari Hive View. This is another option to build and debug HiveQL other than in Visual Studio with the Azure Data Lake Tools plugin. In my blog article Populating data into hive tables, I demonstrated populating internal and external hive …
Continue reading Query Hive Tables with Ambari Hive Views in HDInsight
Blog Series Azure Search Overview Pushing Content To An Index with the .NET SDK I hold the opinion that for a robust indexing strategy, you would likely end up writing a custom batch application between your desired data sources and your defined Azure Search index. The pull method currently only supports data sources that reside …
Continue reading Azure Search: Pushing Content to an Index with the .NET SDK.
Blog Series Azure Search Overview Pushing Content To An Index with the .NET SDK Azure Search is a platform-as-a-service offering. This requires code and configuration to set up and use. Applicable corporate scenarios Enterprise search on many repositories of data or files that are intended to be available for a wide audience. A lightweight one-stop …
Roy Kim on Azure and AI 