Facial Recognition using the UWP APIs in Windows 10 via Windows Forms on .NET Core 3.0

NOTE: The full code for the app built in this post is available on GitHub: https://github.com/robertwray/FacesOfUWP

In a recent post I talked about using the Microsoft.Windows.SDK.Contracts NuGet package to access the UWP APIs provided by Windows 10 in a .NET Core 3.0 based console application. One of the APIs I spotted when I was looking through the namespaces exposed by the package was Windows.Media.FaceAnalysis. Sounds interesting, right? I thought it would be worth seeing if this API is usable (as not all are) from a desktop app and if it was, how complicated it is to use.

It turns out the answer is that it is available and it's pretty easy to use! 

First things first, I'm going to create the project and solution structure with the dotnet command:

md FacesOfUWP
cd FacesOfUWP
dotnet new sln
md FacesOfUWP.UI
cd FacesOfUWP.UI
dotnet new winforms
dotnet sln ..\FacesOfUWP.sln add FacesOfUWP.UI.csproj
dotnet add package Microsoft.Windows.SDK.Contracts -v 10.0.18362.2002-preview

Designing .NET Core 3.0 Windows Forms in Visual Studio

Now that the solution and project are setup, open them up in Visual Studio and run the project, just to make sure that everything's playing nicely. The app compiles and displays a rather empty looking Form1, so now it's time to hook-up some of the UWP facial recognition goodness! 

As of Visual Studio 2019 16.0 and .NET Core 3.0 Preview 5 there isn't any designer support for Windows Forms on .NET Core. A work-around for this, as described in this devblogs.microsoft.com post is to include a .NET Framework based Windows Forms project in your solution and share the code files between them. That's what I'm going to do, which I've achieved by following these steps:

1. Adding a new Windows Forms (.NET Framework) based project called FacesOfUWP.UI.NetFramework to the solution
2. Deleting the auto-generated Form1 from the new project
3. Right-clicking on the .NET Framework based project in Solution Explroer and choosing Add > Existing Item...
4. Navigating to the FacesOfUWP.UI project and selecting Form1.cs from there
5. Clicking on the downward facing button at the right of the Add button and choosing Add As Link to reference the form in its original location rather than take a copy

Once Visual Studio has finished updating the solution it should look like this:

Now the form from the .NET Core project is referenced in the .NET Framework project it can be opened by the Windows Forms designer which makes it a lot easier to work with, so it's time to add a couiple of controls to the form to set it up for working with:

• A button to trigger a File Open Dialog
• The File Open Dialog that's going to be triggered by the button
• A label to display the number of faces found
• A picture box to display the first face, if any, that is found in the selected image

The form should end up looking a little bit like this:

Once it's all setup, make sure that everything's saved then compile and run the .NET Core project - up should pop the Windows Forms form that has been designed in Visual Studio via the built in designer for .NET Framework.

Using the Facial Recognition UWP APIs

Because there aren't (at the time of writing) any non-preview releases of the Microsoft.Windows.SDK.Contracts package, I've had to provide a specific version to reference as without that the output from dotnet add package looks a little bit like this:

  Writing C:\Users\robertwray\AppData\Local\Temp\tmpA763.tmp
info : Adding PackageReference for package 'Microsoft.Windows.SDK.Contracts' into project 'D:\Git\FacesOfUWP\FacesOfUWP.UI\FacesOfUWP.UI.csproj'.
info : Restoring packages for D:\Git\FacesOfUWP\FacesOfUWP.UI\FacesOfUWP.UI.csproj...
...
...
error: Unable to find a stable package Microsoft.Windows.SDK.Contracts with version
error:   - Found 6 version(s) in nuget.org [ Nearest version: 10.0.18362.2002-preview ]
error:   - Found 0 version(s) in Microsoft Visual Studio Offline Packages
error:   - Found 0 version(s) in https://dotnetfeed.blob.core.windows.net/aspnet-aspnetcore-tooling/index.json
error:   - Found 0 version(s) in https://dotnetfeed.blob.core.windows.net/aspnet-aspnetcore/index.json
error:   - Found 0 version(s) in https://dotnetfeed.blob.core.windows.net/aspnet-entityframeworkcore/index.json
error:   - Found 0 version(s) in https://dotnetfeed.blob.core.windows.net/aspnet-extensions/index.json
error:   - Found 0 version(s) in https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json
error:   - Found 0 version(s) in https://dotnetfeed.blob.core.windows.net/dotnet-windowsdesktop/index.json
error: Package 'Microsoft.Windows.SDK.Contracts' is incompatible with 'all' frameworks in project 'D:\Git\FacesOfUWP\FacesOfUWP.UI\FacesOfUWP.UI.csproj'.

You could just as easily add the package inside Visual Studio, I happened to have the console window open still and tend towards using the keyboard when I can.

Next I'm going to add a class to the .NET Core project that'll be responsible for making calls to the Facial Recognition UWP APIs, which I'm going to give the excitingly predictable name of FacialRecognition. Keeping all the code that touches UWP in a separate class will make it easier to ensure that the project compiles in the .NET Core project and the .NET Framework project so I can keep using the designer in the .NET Framework project if I need to tweak the layout of the form in any way.

The last step here is adding a conditional compilation symbol to the .NET Core project that I can hide calls to the FacialRecognition class behind, to prevent compiler errors in the .NET Framework project due to it not being present:

With that added, it's time to start writing some code - starting by double clicking on the Choose... button in the Windows Forms designer to wire-up an event handler that's triggered when its clicked. The code in here is going to be quite simple, no error checking or validation involved:

private void BtnChooseImage_Click(object sender, EventArgs e)
{
    openFileDialog1.Filter = "PNG Files (*.png)|*.png";
    openFileDialog1.ShowDialog();

    var fileStream = openFileDialog1.OpenFile();
}

That's just enough to display the Open File dialog and then retrieve a stream that points to the file that the user selected.

Now that's done it's time to add some code to the method, wrapped in the conditional compilation symbol, that does some facial recognition:

#if DOTNETCORE
var facialRecognition = new FacialRecognition();
facialRecognition.Recognise(fileStream);
#endif

Compiling the whole solution gives one error, from the .NET Core project, of 'FacialRecognition' does not contain a definition for 'Recognise' because the method hasn't been added to the FacialRecognition class yet, so lets get on and do that by adding a using directive for System.IO to the FacialRecognition class and an async void Recognise(Stream fileStream) method, the return type will be added later. With that out of the way, everything compiles in both projects.

The UWP APIs don't use .NET Stream objects so the first thing to do is convert the stream to a Windows.Storage.Streams.IRandomAccessStream by calling the AsRandomAccessStream extension method that's added to the Stream class by System.Runtime.WindowsRuntime (that assembly is referenced by Microsoft.Windows.SDK.Contracts so it should already be available). Then it's a matter of retrieving a SoftwareBitmap from the stream for the facial recognition APIs to use (which will need a using directive for the Windows.Graphics.Imaging namespace). The Recognise method should now look like this:

public async void Recognise(Stream fileStream)
{
    var randomAccessStream = fileStream.AsRandomAccessStream();

    var bitmapDecoder = await BitmapDecoder.CreateAsync(randomAccessStream);
    var rawBitmap = await bitmapDecoder.GetSoftwareBitmapAsync();
}

That's all the preamble that's required to get a bitmap, but not quite enough to get one that the Facial Recognition API will work with. In order to give the Facial Recognition APIs an image in a format they're happy with, the rawBitmap needs to be converted into one of the supported formats:

var supportedBitmapFormats = FaceDetector.GetSupportedBitmapPixelFormats();
var supportedFormatBitmap = SoftwareBitmap.Convert(rawBitmap, supportedBitmapFormats.First());

The FaceDetector class comes from the Windows.Media.FaceAnalysis namespace, we could probably check to see if the supported formats contains the format that rawBitmap is already in and skip conversion if that's the case but for simplicities sake I'm skipping doing that for now. If this was anything other than a proof of concept though, definitely one to consider!

Actually detecting faces in the bitmap is a bit of anti-climax, consisting of these two lines of code:

var faceDetector = await FaceDetector.CreateAsync();
var faces = await faceDetector.DetectFacesAsync(supportedFormatBitmap);

I'm going to start the process of returning the recognised faces by simply changing the return type of the method to Task<int> and returning faces.Count() at the end of the method. Finally I'm going to change the button click handler so it's async and update the content of the #if DOTNETCORE so it takes the result of calling Recognise and drops it into the label on the form:

#if DOTNETCORE
var facialRecognition = new FacialRecognition();
var faces = await facialRecognition.Recognise(fileStream);
lblNumberOfFaces.Text = string.Format("The number of faces found is: {0}", faces);
#endif

There's enough code there now to take an image, look for faces and return the number of them that are found! Here's the result of running the image on the right-hand side of my 'About Me' page through the app:

And here's the image, to save you going to look for it:

Extracting faces

The very last step is to get the face that's been recognised and drop it into the PictureBox. First up I'm going to declare a type to use to return the face count and face from the Recognise method:

public class RecogniseResult
{
    public int Faces { get; set; }
    public Stream FirstFace { get; set; }
}

With that available, it's a fairly simple matter (again - this is bare minimum code, there's no error checking or validation and assumptions about image sizes and formatting have been made!) to extract the recognised face and return it as a Stream:

var result = new RecogniseResult();

if (faces.Any())
{
    result.Faces = faces.Count();

    var memoryStream = new InMemoryRandomAccessStream();

    var bitmapEncoder = await BitmapEncoder.CreateAsync(BitmapEncoder.BmpEncoderId, memoryStream);
    bitmapEncoder.SetSoftwareBitmap(rawBitmap);
    bitmapEncoder.BitmapTransform.Bounds = faces.First().FaceBox;

    await bitmapEncoder.FlushAsync();

    result.FirstFace = memoryStream.AsStream();
}

return result;

That stream can then be used inside the #ifdef:

var facialRecognition = new FacialRecognition();
var faces = await facialRecognition.Recognise(fileStream);
lblNumberOfFaces.Text = string.Format("The number of faces found is: {0}", faces.Faces);
pictureBox1.Image = Image.FromStream(faces.FirstFace);

And here we have the result:

It's worth mentioning that some of the UWP image APIs are a little fussy and I've seen exceptions when using images where the X/Y are odd numbers, don't ask me why! I solved this by using pictures with even dimensions - that's enough for a proof of concept!