{"id":3731,"date":"2017-07-05T00:00:00","date_gmt":"2017-07-05T00:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse"},"modified":"2025-06-20T07:23:53","modified_gmt":"2025-06-20T14:23:53","slug":"handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/","title":{"rendered":"Handling data encoding issues while loading data to SQL Data Warehouse"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">This blog is intended to provide insight on some of the data encoding issues that you may encounter while using Polybase to load data to SQL Data Warehouse. This article also provides some options that you can use to overcome such issues and load the data successfully.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"problem\">Problem<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In most cases, you will be migrating data from an external system to <a href=\"https:\/\/azure.microsoft.com\/en-us\/resources\/cloud-computing-dictionary\/what-is-a-data-warehouse\" target=\"_blank\" rel=\"noopener\">SQL Data Warehouse<\/a> or working with data that has been exported in flat file format. If the data is formatted using either the UTF-8 or UTF-16 encoding standard, you can use Polybase to load the data. However, the format of your data is dependent on the encoding options supported by the source system. Some systems do not provide support for UTF-8 or UTF-16 encoding. If the data you are working with is formatted in an alternate format, such as ISO-8859-1, then being able to convert the data to UTF-8\/UTF-16 format can save valuable time and effort.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The flow of data from a source system to Azure Blob Storage and then on to Azure SQL Data Warehouse (DW) is shown in the following graphic:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/963a1ce4-977f-4ec7-84fd-04496cd8bc0e.webp\" alt=\"Azure Blog Storage\" title=\"Azure Blog Storage\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Azure Blob Storage is a convenient place to store data for use by Azure services like SQL DW. PolyBase makes it easy to access the data by using T-SQL, for example creating external tables for the data on Azure Blob Storage and loading the data into internal tables of SQL Data Warehouse using a simple SELECT query.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If the volume of the data being loaded is small, then it may be easier to export the data from the source system again, this time using UTF-8\/UTF-16 encoding. For larger volumes of data, however, re-export, data compression, and data load to Azure Blob Storage can take weeks. To avoid this delay, you need to be able to convert the encoding on the data files within the Azure environment without accessing the source system again.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"solution\">Solution<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The sections below provides details on options you have for converting source file encoding to UTF-8\/UTF-16.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><b>Important:<\/b> PolyBase supports UTF16-LE. It shouldn\u2019t matter for customers in the Windows ecosystem, but a customer may specify UTF16-BE and have their load fail.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"option-1-notepad\">Option 1: Notepad++<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">You can use the Notepad++ tool to change the encoding of a file on a local computer. Simply download the data file to a local computer, open the file in Notepad++, and then convert the file encoding to UTF-8\/UTF-16.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">1. To <b>view the encoding of a source file<\/b>, click the <b>Encoding<\/b> menu, as shown in the following graphic:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/7af74fd8-2347-45ee-bfac-2a885ae4bd05.webp\" alt=\"Encode 1\" title=\"Encode 1\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The source file in the example above is encoded in ANSI.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">2. To<b> convert file encoding to UTF-8<\/b>, on the <b>Encoding<\/b> menu, select <b>Convert to UTF-8<\/b>.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/a9eb1ddb-130b-4665-bc9b-6ad47c6f2fb6.webp\" alt=\"Encode 2 \" title=\"Encode 2 \" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">3. Save the file, use the <b>Encoding<\/b> menu to view the encoding, and confirm that the file is now encoded using UTF-8.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/35f83cbe-6d12-4370-ab2e-a5ede733d20e.webp\" alt=\"Encode 3\" title=\"Encode 3\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">After the file is saved in UTF-8 encoding, you can use Polybase to upload it to Azure Blob Storage and load it into SQL Data Warehouse.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">While this is a viable approach, there are some drawbacks, which are listed below:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">Download time<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Available space on local system<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Upload time<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Works only with small files because of memory and space constraints<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"option-2-azure-vm\">Option 2: Azure VM<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To overcome some of the drawbacks associated with using Notepad++, you can use an Azure VM to convert data file encoding. With this method, the entire process occurs within the Azure environment, thereby eliminating delays associated with transferring data between Azure and the local system. This process is shown in the following graphic:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/2f34251c-63e1-4b88-8dc4-abb74b59d24d.webp\" alt=\"Azure VM\" title=\"Azure VM\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">This approach has the following high-level steps:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">Setup an Azure VM (Windows or Linux)<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Download data file from Azure Blob Storage to local storage on Azure VM<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Extract data file (if applicable)<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Convert data file encoding using a utility (custom\/built-in)<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Upload the converted data file from local storage on Azure VM to Azure Blob Storage<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Note that this approach has its own drawbacks:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">Download time<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Available space on local system<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Upload time<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"option-3-azure-file-storage\">Option 3: Azure File Storage<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To overcome the limitations associated with download and upload time when using Azure VMs, you can use Azure File Storage, which offers cloud-based SMB file shares that you can use to quickly migrate legacy applications that rely on file shares to Azure without costly rewrites. With Azure File Storage, applications running in Azure virtual machines or cloud services can mount a file share in the cloud, just as a desktop application mounts a typical SMB share. Any number of application components can then mount and access the File Storage share simultaneously, as shown in the following graphic:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/410f33c3-4423-497c-8bac-93a9edf1a50c.webp\" alt=\"Azure Storage Concepts \" title=\"Azure Storage Concepts \" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Note: <\/strong>Learn more about <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/storage\/storage-introduction\">Azure Storage<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When using Azure File Storage, be aware of the capacity limits identified in the following table:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/e55e5b18-172e-47d9-80b2-18171a9806fe.webp\" alt=\"Resource-Limit\" title=\"Resource-Limit\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Note:<\/strong> A full listing of <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/storage\/storage-scalability-targets\">Azure Storage Scalability and Performance Targets<\/a> is now available.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With this approach, you can have all the data files on Azure File Storage and have an Azure VM that can mount Azure File Storage. After having the mount, the Azure VM can directly read and write files from\/to Azure File Storage without having to download to or upload from local storage on Azure VM.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This approach includes the following high-level steps:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">Setup an Azure VM (Windows or Linux)<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Mount Azure File Storage on Azure VM (see procedure below)<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Extract data file (if applicable)<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Convert data file encoding using a utility (custom\/built-in)<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">The diagram below shows the complete flow of data compression, transfer, extraction, transformation, and load via PolyBase into SQL DW:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/abcc89a0-d77b-4166-9217-934172516e6a.webp\" alt=\"image\" title=\"image\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"mounting-azure-file-storage-to-vm\">Mounting Azure File Storage to VM<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The process of mounting Azure File Storage to VM, Ubuntu Linux VM in this case, involves three high-level steps:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">Installing the required libraries\/packages.\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">sudo apt-get install <b>cifs-utils<\/b><\/li>\n<\/ul>\n<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Creating the mount point location on Azure VM to which the Azure File Storage will be mapped.\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">sudo mkdir \/mnt\/mountpoint<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Mounting Azure File Storage location to Azure VM mount point.\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">sudo mount -t cifs <b>\/\/myaccountname.file.core.windows.net\/mysharename<\/b> <b>\/mnt\/mountpoint<\/b> -o <b>vers=3.0<\/b>,user=myaccountname,password=StorageAccountKeyEndingIn==,dir_mode=0777,file_mode=0777,serverino<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\"><b>Note<\/b>: Get full details on <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/storage\/storage-how-to-use-files-linux\">mounting Azure File Storage from a Linux VM<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"automating-data-encoding-conversion\">Automating data encoding conversion<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">This section provides some details on a project that leveraged this approach to convert the encoding of a data file:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">131 tables data exported from Netezza system<\/li>\n\n\n\n<li class=\"wp-block-list-item\">4 data files per source table organized under the folder name representing the source table<\/li>\n\n\n\n<li class=\"wp-block-list-item\">All data files encoded in ANSI format (ISO-8859-1)<\/li>\n\n\n\n<li class=\"wp-block-list-item\">All data files compressed using GZ compression<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Total compressed data files size was <b>750GB<\/b><\/li>\n\n\n\n<li class=\"wp-block-list-item\">Total uncompressed converted data files size was <b>7.6TB<\/b><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The data files were organized on Azure File Storage in the following structure:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/53c86540-0f4a-471f-886b-fae16a51eb98.webp\" alt=\"ConvertedData\" title=\"ConvertedData\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">A snapshot of the bash script on Ubuntu VM that was used to convert the encoding on the data files automatically is shown in the following graphic:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/88b80df0-6220-4b11-96ac-52cd1421e5bf.webp\" alt=\"3d8849cb-bc90-4558-88a0-212b2d2f7be4\" title=\"3d8849cb-bc90-4558-88a0-212b2d2f7be4\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">This script performed the following:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">Accepted the table name as an argument<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Looped through each of the 4 data files for the given table<\/li>\n\n\n\n<li class=\"wp-block-list-item\">For each data file\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">Extracted the compressed GZ file using <b>gunzip<\/b> command<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Converted the encoding of each file using <b>iconv<\/b> command where the source file encoding is specified as ISO-8859-1 and the target file encoding is specified as UTF-8<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Wrote the converted file to a folder with the table name under <b>ConvertedData<\/b><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">The script was further enhanced to loop through a list of table names and repeat the above process, rather than accepting the table name as an argument.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"convert-from-any-encoding-to-any-other-encoding\">Convert from any encoding to any other encoding<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">The script can be modified to accept the <b>from<\/b> and <b>to<\/b> encoding as arguments instead of hardcoding them in the script. A full list of encodings supported by iconv command can be retrieved by running the command <b><i>iconv -l<\/i><\/b> on the computer you will be using to convert the data encoding. Be sure to check for any typos in the encoding format specified before running the command. A snapshot of the generic script and an example on how to invoke it is shown in the following graphic:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/71ed8152-a4d5-41d5-bd81-b242cc95e415.webp\" alt=\"afa49072-d4d6-4918-8b99-e46740002686\" title=\"afa49072-d4d6-4918-8b99-e46740002686\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/37613f81-1dfe-471e-8954-481dd706b1fd.webp\" alt=\"a33935b2-bb3a-4185-ac81-af6f975cc934\" title=\"a33935b2-bb3a-4185-ac81-af6f975cc934\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The above command converts the data files from UTF-8 encoding to ISO_8859-1 encoding format.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"recognition\">Recognition<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The Data Migration Team would like to thank primary contributors Rakesh Davanum, Andy Isley, Joe Yong, Casey Karst, and Mukesh Kumar, for their efforts in preparing this blog posting. The details provided has been harvested as part of a customer engagement sponsored by the CSE DM Jumpstart Program.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This blog is intended to provide insights on some of the data encoding issues that you may encounter while using Polybase to load data to SQL Data Warehouse.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ms_queue_id":[],"ep_exclude_from_search":false,"_classifai_error":"","_classifai_text_to_speech_error":"","_alt_title":"","footnotes":"","msx_community_cta_settings":[]},"categories":[1473,1466],"tags":[],"audience":[3057,3055,3053,3056],"content-type":[1511],"product":[1539],"tech-community":[],"topic":[],"coauthors":[906],"class_list":["post-3731","post","type-post","status-publish","format-standard","hentry","category-databases","category-hybrid-multicloud","audience-data-professionals","audience-developers","audience-it-decision-makers","audience-it-implementors","content-type-best-practices","product-azure-sql-database","review-flag-1680286581-295","review-flag-1680286581-364","review-flag-1-1680286581-825","review-flag-2-1680286581-601","review-flag-3-1680286581-173","review-flag-4-1680286581-250","review-flag-7-1680286581-146","review-flag-8-1680286581-263","review-flag-vm-1680286585-143"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Handling data encoding issues while loading data to SQL Data Warehouse | Microsoft Azure Blog<\/title>\n<meta name=\"description\" content=\"This blog is intended to provide insights on some of the data encoding issues that you may encounter while using Polybase to load data to SQL Data Warehouse.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Handling data encoding issues while loading data to SQL Data Warehouse | Microsoft Azure Blog\" \/>\n<meta property=\"og:description\" content=\"This blog is intended to provide insights on some of the data encoding issues that you may encounter while using Polybase to load data to SQL Data Warehouse.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/\" \/>\n<meta property=\"og:site_name\" content=\"Microsoft Azure Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/microsoftazure\" \/>\n<meta property=\"article:published_time\" content=\"2017-07-05T00:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-20T14:23:53+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/963a1ce4-977f-4ec7-84fd-04496cd8bc0e.webp\" \/>\n<meta name=\"author\" content=\"Matt Goswell\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@azure\" \/>\n<meta name=\"twitter:site\" content=\"@azure\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Matt Goswell\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/\"},\"author\":[{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/matt-goswell\/\",\"@type\":\"Person\",\"@name\":\"Matt Goswell\"}],\"headline\":\"Handling data encoding issues while loading data to SQL Data Warehouse\",\"datePublished\":\"2017-07-05T00:00:00+00:00\",\"dateModified\":\"2025-06-20T14:23:53+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/\"},\"wordCount\":1399,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/963a1ce4-977f-4ec7-84fd-04496cd8bc0e.webp\",\"articleSection\":[\"Databases\",\"Hybrid + multicloud\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/\",\"name\":\"Handling data encoding issues while loading data to SQL Data Warehouse | Microsoft Azure Blog\",\"isPartOf\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/963a1ce4-977f-4ec7-84fd-04496cd8bc0e.webp\",\"datePublished\":\"2017-07-05T00:00:00+00:00\",\"dateModified\":\"2025-06-20T14:23:53+00:00\",\"description\":\"This blog is intended to provide insights on some of the data encoding issues that you may encounter while using Polybase to load data to SQL Data Warehouse.\",\"breadcrumb\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#primaryimage\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/963a1ce4-977f-4ec7-84fd-04496cd8bc0e.webp\",\"contentUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/963a1ce4-977f-4ec7-84fd-04496cd8bc0e.webp\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Blog home\",\"item\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Databases\",\"item\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/category\/databases\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Handling data encoding issues while loading data to SQL Data Warehouse\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#website\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\",\"name\":\"Microsoft Azure Blog\",\"description\":\"Get the latest Azure news, updates, and announcements from the Azure blog. From product updates to hot topics, hear from the Azure experts.\",\"publisher\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\",\"name\":\"Microsoft Azure Blog\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp\",\"contentUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp\",\"width\":512,\"height\":512,\"caption\":\"Microsoft Azure Blog\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/microsoftazure\",\"https:\/\/x.com\/azure\",\"https:\/\/www.instagram.com\/microsoftdeveloper\/\",\"https:\/\/www.linkedin.com\/company\/16188386\",\"https:\/\/www.youtube.com\/user\/windowsazure\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/person\/c702e5edd662b328b49b7e1180cab117\",\"name\":\"shakir\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g7664e653ea371ce16eaf75e9fa8952c4\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g\",\"caption\":\"shakir\"},\"sameAs\":[\"https:\/\/azure.microsoft.com\"],\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/shakir\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Handling data encoding issues while loading data to SQL Data Warehouse | Microsoft Azure Blog","description":"This blog is intended to provide insights on some of the data encoding issues that you may encounter while using Polybase to load data to SQL Data Warehouse.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/","og_locale":"en_US","og_type":"article","og_title":"Handling data encoding issues while loading data to SQL Data Warehouse | Microsoft Azure Blog","og_description":"This blog is intended to provide insights on some of the data encoding issues that you may encounter while using Polybase to load data to SQL Data Warehouse.","og_url":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/","og_site_name":"Microsoft Azure Blog","article_publisher":"https:\/\/www.facebook.com\/microsoftazure","article_published_time":"2017-07-05T00:00:00+00:00","article_modified_time":"2025-06-20T14:23:53+00:00","og_image":[{"url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/963a1ce4-977f-4ec7-84fd-04496cd8bc0e.webp","type":"","width":"","height":""}],"author":"Matt Goswell","twitter_card":"summary_large_image","twitter_creator":"@azure","twitter_site":"@azure","twitter_misc":{"Written by":"Matt Goswell","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#article","isPartOf":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/"},"author":[{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/matt-goswell\/","@type":"Person","@name":"Matt Goswell"}],"headline":"Handling data encoding issues while loading data to SQL Data Warehouse","datePublished":"2017-07-05T00:00:00+00:00","dateModified":"2025-06-20T14:23:53+00:00","mainEntityOfPage":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/"},"wordCount":1399,"commentCount":0,"publisher":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#primaryimage"},"thumbnailUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/963a1ce4-977f-4ec7-84fd-04496cd8bc0e.webp","articleSection":["Databases","Hybrid + multicloud"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/","name":"Handling data encoding issues while loading data to SQL Data Warehouse | Microsoft Azure Blog","isPartOf":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#primaryimage"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#primaryimage"},"thumbnailUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/963a1ce4-977f-4ec7-84fd-04496cd8bc0e.webp","datePublished":"2017-07-05T00:00:00+00:00","dateModified":"2025-06-20T14:23:53+00:00","description":"This blog is intended to provide insights on some of the data encoding issues that you may encounter while using Polybase to load data to SQL Data Warehouse.","breadcrumb":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#primaryimage","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/963a1ce4-977f-4ec7-84fd-04496cd8bc0e.webp","contentUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/07\/963a1ce4-977f-4ec7-84fd-04496cd8bc0e.webp"},{"@type":"BreadcrumbList","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/handling-data-encoding-issues-while-loading-data-to-sql-data-warehouse\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog home","item":"https:\/\/azure.microsoft.com\/en-us\/blog\/"},{"@type":"ListItem","position":2,"name":"Databases","item":"https:\/\/azure.microsoft.com\/en-us\/blog\/category\/databases\/"},{"@type":"ListItem","position":3,"name":"Handling data encoding issues while loading data to SQL Data Warehouse"}]},{"@type":"WebSite","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#website","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/","name":"Microsoft Azure Blog","description":"Get the latest Azure news, updates, and announcements from the Azure blog. From product updates to hot topics, hear from the Azure experts.","publisher":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/azure.microsoft.com\/en-us\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization","name":"Microsoft Azure Blog","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp","contentUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp","width":512,"height":512,"caption":"Microsoft Azure Blog"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/microsoftazure","https:\/\/x.com\/azure","https:\/\/www.instagram.com\/microsoftdeveloper\/","https:\/\/www.linkedin.com\/company\/16188386","https:\/\/www.youtube.com\/user\/windowsazure"]},{"@type":"Person","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/person\/c702e5edd662b328b49b7e1180cab117","name":"shakir","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g7664e653ea371ce16eaf75e9fa8952c4","url":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g","caption":"shakir"},"sameAs":["https:\/\/azure.microsoft.com"],"url":"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/shakir\/"}]}},"msxcm_display_generated_audio":false,"msxcm_animated_featured_image":null,"distributor_meta":false,"distributor_terms":false,"distributor_media":false,"distributor_original_site_name":"Microsoft Azure Blog","distributor_original_site_url":"https:\/\/azure.microsoft.com\/en-us\/blog","push-errors":false,"_links":{"self":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/3731","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/comments?post=3731"}],"version-history":[{"count":1,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/3731\/revisions"}],"predecessor-version":[{"id":42692,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/3731\/revisions\/42692"}],"wp:attachment":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/media?parent=3731"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/categories?post=3731"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/tags?post=3731"},{"taxonomy":"audience","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/audience?post=3731"},{"taxonomy":"content-type","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/content-type?post=3731"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/product?post=3731"},{"taxonomy":"tech-community","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/tech-community?post=3731"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/topic?post=3731"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/coauthors?post=3731"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}