Converting a load of files from UTF-16 to UTF-8

A sure sign that you've scripted a load of stuff out as UTF-16 and are going to have to go through and do it all over again

There's a database here that I've been meaning to get into source control for a long time. Luckily we have regular backups of it (that have been restore proven) and it never changes so it's been pretty low down the priority list. Along comes today and I need to make some changes so, time to get it all scripted. I do this using SQL Server Management Studio, scripting one object per file and dropping each object type into separate folders. I usually remember to script it out as 'ANSI Text' but this time round I forgot, hence GitHub Desktop showing me this dreaded message.

For various reasons scripting this database is a convoluted exercise, meaning that this one small mistake meant it would take me the best part of an hour to do again. Enter C# and a teeny tiny .NET Core console app (though equally valid as a .NET Framework app as it's not using anything .NET Core specific) to re-save all the files. Without further ado, should the code be useful to you, here it is:

using System;
using System.IO;
using System.Text;

namespace ConvertToUTF8
{
    class Program
    {
        static void Main(string[] args)
        {
            var folder = args[0];

            var files = Directory.GetFiles(folder, "*.sql", SearchOption.AllDirectories);

            foreach (var file in files)
            {
                Console.WriteLine($"Processing {file}");
                var content = File.ReadAllText(file);

                File.WriteAllText(file, content, Encoding.UTF8);
            }
        }
    }
}

This iterates over every .sql file in the folder passed in as the first argument to the app, and its sub-folders, and re-saves the file as UTF-8 instead of UTF-16. Five minutes to write the app, 10 seconds for it to run and that means the best part of an hour saved. Awesome!

About Rob

I've been interested in computing since the day my Dad purchased his first business PC (an Amstrad PC 1640 for anyone interested) which introduced me to MS-DOS batch programming and BASIC.

My skillset has matured somewhat since then, which you'll probably see from the posts here. You can read a bit more about me on the about page of the site, or check out some of the other posts on my areas of interest.

No Comments

Add a Comment