TinyDocumentDB - part 3 - querying for data (the plan)

Alright, so this is completely new ground for me. I've never written any kind of dynamic query function for real before. At least not where I also had to design the indexing functionality. I'm writing this part of the test even before I've started to write any code at all since I need to get a grip of the requirements here.

Later on, I do need to add some kind of schema support to divide different types of entities from each other. At the moment, I'm just going to ignore that fact and treat the data as a single entity. From a tech perspective, it's the same thing if it's one or multiple entities in the database.

The plan for upserting data

Let's start out with a simple definition of what we should accomplish:

• As data flow into the document database (as an INSERT or as an UPDATE, knowns as an  UPSERT), we need to pick up the values of certain fields and store them in some kind of index with a reference back to the original data. (hmm, what about a feature to be able to search historical data).
• When we search for data, we will query this index and return results based on that.
• It will not be optimized in any way, just make it work.

Straight forward enough.

The diagram

A plan is nothing without a diagram.

The step-by-step plan for storing data (post diagram)

After the diagram, we have enough to create a step-by-step plan:

1. On the left-hand side, we have data, we pass that data to the core-client.
2. Store the data as fast as possible in storage.
3. The core-client reads any index definition associated with the type of data (we only have one type... data...) and extracts the data from the fields that we define in the index definition.
4. Read the index-file from storage, update it, and write it to storage.

Of course, from a perspective point of view, this will perform terribly under pressure. But we'll get to that later on.

The plan for querying data

I admit, I haven't given the query language details any thought at all and it will not look like this later on. 

But I do think that it's important to have a GET query mechanism available as well as a POST query mechanism. The GET variant will be easier to use ad-hoc and the POST can offer some more fine-grained usage.

 

The step-by-step plan for querying data (post diagram)

At a top-level, we would do something like this.

1. Get the query to the Core.Client.
2. Read the associated index file based on the entity type (we only have one)
3. Parse the index file for search results and store any unique document keys (the index file could also contain more information on where in the document the information is found and such perhaps?).
4. Read all the data from storage and return it.

Straightforward enough. Now I have a rough plan to get started with the implementation.

Summary

So achieved so far: (blog post part number within [x])

• [2] A REST API for reading and writing
• [2] InMemory or File-system reader writer
• [2] CI/CD pipeline using Github Actions
• [3] A plan for indexing data and for querying it

New ideas/questions:

• [2] How to handle schemas without making it complicated?
• [2] Schema stitching?
• [3] Storing more data in the index?

Next up:

• Querying of data - simple indexing implementation

TinyDocumentDB - Part 2 - The most simple API ever

So this is what I got so far. On the left side, we have a REST API with support for GET using an Id and two POST implementations.

One of the POST implementations needs an explicitly defined Id in the route, the other will use reflection to get the id from the content. At the moment, it just looks for a property named 'Id', but it should be definable later on if this is something that will stand the test of time.


The Core.Client is the coordinator at the moment for reading and writing data to some kind of storage. I've implemented two basic variants; an InMemoryReader/writer and a FileReader/Writer.

The API

Created as a .NET Core WebApi that serves as a runtime host for the Core.Client basically.

I only handle JSON at the moment and plan to keep it that way.

The reader and the writers

The reader and writer should simply read and store data to some kind of storage as fast as possible. It should not handle caching or any hocus pocus at all. (we will get to caching later on).

  public interface IStorageReader
  {
      Task<string> Read(string id);
  }

  public interface IStorageWriter
  {
      Task Write(string id, string content);
      Task Delete(string id);
  }

To illustrate how basic this is right now, I present the Write() method of the FileStorageWriter to you:

  public async Task Write(string id, string content)
  {
      var path = Path.Combine(FileSettings.BaseFolder, FileSettings.GenerateFileNameFromId(id));
      Directory.CreateDirectory(FileSettings.BaseFolder);
      await System.IO.File.WriteAllTextAsync(path, content);
  }

This code creates a path to the file by appending a base folder (hardcoded) and a filename generated from the SHA256 hash of the Id passed into the function. Then we write all the content to disk.

Summary

So achieved so far:

• A REST API for reading and writing
• InMemory or File-system reader writer
• CI/CD pipeline using Github Actions

New ideas/questions:

• How to handle schemas without making it complicated?
• Schema stitching?

Next up:

• Querying of data - simple indexing strategy

A Tiny DocumentDB-as-a-Service

It's been a while since I've blogged. About 2,5 years to be exact. Historically, this blog was all about Xamarin but I've decided to expand the surface area to just about anything regarding coding, electronics, and development. There will still be a lot of Xamarin of course.

In my current assignment, I've been doing a lot of backend development and have stumbled across many different data sources.

I must admit that I haven't worked with document databases before and honestly thought that all of this NoSQL-stuff was kinda weird. Now, I've changed my mind and also included document databases into my container of stuff that I love.

So, to pay tribute to this new love, I've decided to create the tiniest version of a document database as a service that I possibly can do and I'll add it to the TinyStuff collection of tiny stuff on Github in due time.

I will from time to time post about my progress here and of course, the code will be available for anyone to use.

The naive blue-print

TinyDocumentDb must (should?) support (no order of priority):

• REST API
• .NET Core WebAPI (and/or Azure Functions deploy) runtime
• Caching
• Extendable
• Storage independent (think file system, Azure Blob Storage, permanent marker on a duck)
• Partial document updates
• Eventual consistency
• Secure

So that's that for this post. The next will most likely a very simple sample API.

Visual Studio for Mac installer crashes

There are some reports of VS for Mac crashing during install. I got to experience this first hand this weekend. The solution for me was to fire up the terminal and run the installer using sudo.

Step 1

Given that you have the installer image mounted and a terminal running.
Navigate to /volumes/Install Visual Studio for Mac.app/contents/MacOS

Step 2

Run "sudo ./Install_Visual_Studio"

Android 8 and condition `ji' not met

A quick tips/workaround. I'm sure Xamarin will address this issue shortly.

Problem

The app crashes on start up on Android 8 with the error message

[] * Assertion at /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mono/mini/mini-arm64.c:878, condition `ji' not met

Updated solution (the real solution)

It was actually the usage of the dynamic keyword that caused the crash...

Solution (kept for historical reason)

Uncheck "Use Shared Mono Runtime" under "Android Build".

WebView blank in Xamarin Forms on UWP

TLDR;

If you are hooking up events for a Xamarin Forms WebView, wrap the hooking up in a Device.BeginInvokeOnMainThread if you experience a blank WebView. I've seen this issue mostly on UWP.

    public MyView()
    {
        this.InitializeComponent();
       
        Device.BeginInvokeOnMainThread(() =>
        {

            // Hook up events in here
            this.myWebView.Navigating += async (s, e) =>
            {
                // Do stuff here
            };
        });
    }

Longer version

I have a login view in my app and the code looks like this.

    public partial class LoginView : ContentPage
    {
        public LoginView(LoginViewModel vm)
        {
            this.InitializeComponent();
            this.BindingContext = vm;

            this.LoginWebView.Navigating += async (s, e) =>
            {
                this.BusyLabel.IsVisible = true;
                await vm?.Navigating(e.Url);
            };

            this.LoginWebView.Navigated += async (s, e) =>
            {
            
                this.BusyLabel.IsVisible = false;
            };  
        }
    }

I needed to hook up the Navigating and Navigated events for two reasons;

1. To display a busy/loading label over the webview
2. To pass the resulting URL back to the ViewModel

This works... Most of the time. But sometimes the WebView just displays a white square. I tried a bunch of different urls trying to determine if it was something in the html that was messing something up. 

Then I read this post (https://forums.xamarin.com/discussion/63280/xamarin-forms-webview-does-not-work-once-the-navigating-eventhandler-is-setup). It wasn't really about this issue but the title of the post made me think.

Perhaps, if I delay the event hooking-uping it might magically work again. 

So I wrapped the hookups in a Device.BeginInvokeOnMainThread(...) call and it just seems to work.

An other related issue you might experience is this. But this goes for all controls, not only WebView.

• https://stackoverflow.com/questions/34330446/xamarin-forms-webview-not-showing-up

Mapping without Automapper

I've tried AutoMapper on numerous occasions and it's a great product. However, my mind seems to work a little bit different so I usually use a little different approach. I want to be able to follow my mappings so what I end up doing is to write the mappers by hand. (#bad)

I do give AutoMapper a try from time to time, but I still favour this way of doing it. It's just a personal preference!

If you haven't heard of Automapper, check it out at http://automapper.org/ before you embrace this technique! :)

The pattern I use for this is simple (and the magic lies in the simplicity of it).

Step 0 - Define the types on each side of the fence

Investigate the types you want to map. I have two ducks.

    public class BackendDuck
    {
        public int Id { get; set; }
        public int BeakSize { get; set; }
        public string TotallySecretDuckKey { get; set; }
    }

    public class FrontEndDuck
    {
        public int Id { get; set; }
        public int BeakSize { get; set; }
    }

Step 1 - Create the mapper class

I create a static Mapper class that will handle my mappings.

   public static partial class Mapper
   {
   }

Why partial? Because I like to keep the option open to create another file to append to this mapper class. The file on disk could be called DuckMapper.cs. Then why isn't the class called DuckMapper.cs? Because we really don't want to check fifteen different classes to find the correct mapper. The answer is within step 2.

Step 2 - create the map methods:

Create two methods, both called Map (this is important), with the signature below.

    public static partial class Mapper
    {
        public static IEnumerable<FrontEndDuck> Map(IEnumerable<BackendDuck> source)
        {
            return null;
        }

        public static FrontEndDuck Map(BackendDuck source)
        {
            return null;
        }
    }

We always want two methods, one to Map an IEnumerable and one to map an item.

Step 3a - implement the map methods (non-linq way):

The first way of implementing the methods is by not using Linq. It's the straight forward, C# way to do it. No surprises at all.

    public static partial class Mapper
    {
        public static IEnumerable<FrontEndDuck> Map(IEnumerable<BackendDuck> source)
        {
            var result = new List<FrontEndDuck>();

            foreach(var item in source)
            {
                result.Add(Map(item));
            }

            return result;
        }

        public static FrontEndDuck Map(BackendDuck source)
        {
            return new FrontEndDuck()
            {
                Id = source.Id,
                BeakSize = source.BeakSize
            };
        }
    }

Step 3b - implement the map method using Linq

This does the same thing, but the looping part is hidden in a .Select extension method.

    public static partial class Mapper
    {
        public static IEnumerable<FrontEndDuck> Map(IEnumerable<BackendDuck> source)
        {
            return source.Select(x => Map(x));
        }

        public static FrontEndDuck Map(BackendDuck source)
        {
            return new FrontEndDuck()
            {
                Id = source.Id,
                BeakSize = source.BeakSize
            };
        }
    }

Step 4 - usage

This is how you would use the Mapper.

    public class DuckManager
    {
        public void DoStuff()
        {
            // Create a single duck
            var backendDuck = new BackendDuck() 
            { 
                Id = 1, 
                BeakSize = 42, 
                TotallySecretDuckKey = "TheKey" 
            };

            // Add the duck to a list
            var backendDucks = new List<BackendDuck>()
            {
                backendDuck
            };

            // Use the mappings - if setup correctly, 
            // all you have to remember is Mapper.Map(...)
            var item = Mapper.Map(backendDuck);
            var list = Mapper.Map(backendDucks);
        }
    }

Summary

Pros:

• It's easy to follow debugging
• You get all your mappings written in C# code that you can follow, nothing is hidden
• Written correctly, just remember Mapping.Map( ) and you are set

Cons:

• More code to write

Resources

Check out Automapper if you haven't done so. It might fit your needs better.