BlockDraw

BlockDraw is a game where the user will be asked to create an object, like, a house, a car, a camera, etc. only using the three basic shapes, i.e., Rectangle, Triangle and Round. Users can reposition those shapes however they like to form the structure of the object; they can even change the orientation, adjust the parameters and resize the shapes to better fit in the said object. Once the object is created, it is verified by a custom Core ML model to check its validity, also providing information like, which and how many shapes were used in creating the object.

BlockDraw is especially for kids, but it can also be a fun thing to play for adults too. For kids and toddlers, this game can be a great source where they can develop understanding of different types of shapes and its parameters that can be modified to make newer shapes. BlockDraw will also help to develop understanding of combining these shapes in some meaningful ways to form and architect different objects.

How to make objects using only three shapes?

Well, that's the challenge. The objects that are asked, can always be created using these three shapes. One has to be creative and think differently on how they can put everything together to form the said object. Here are some tips/features outlined below that can help with that:

1. Decide which shapes to use

There can be some dominating shape which can act as the foundational structure or the frame for a given object. For example, a big rectangle can be a dominating shape for a 'Car'. Likewise, relatively smaller circles for wheels and so on.

2. Positioning shapes

Once having a rough picture of that object in mind, just add shapes on the Canvas and position it relative to each other to form a structure resembling the said object.

3. Adjust some shape-specific parameters

When an object is selected, Side Pane will show various parameters that can be adjusted for that shape to make it a more suitable fit in that object.

Moreover, adjusting some shape-specific parameters may also lead to forming new shapes that can contribute in creating the object. Here's something more about it...

Rectangle: The rectangle offers two adjustable parameters: length and breadth. Adjusting these two can form a square if both are made equal and if one of those is made extremely small, it can form a line with a range of thicknesses. Another parameter that’s common to all shapes is rotation. This parameter, when adjusted to ±90 degrees can form Diamond.

Round: The round shape offers two such parameters, i.e. Vertical Radius and Horizontal Radius. When both are equal, it’s a circle and when not equal, it forms an Oval.

Triangle: The triangle offers two such parameters: base and altitude which can be adjusted to form various different triangles.

4. Give it your touch by tweaking parameters, orientation and positioning

These shapes, when positioned well with proper parameters, can form countless different 2D objects, like house, car, camera and more, and not just with one appearance, but with variety of styles. So it's fun to tweak it a little and play with the created object.

BlockDraw is built entirely using Swift and Apple's native frameworks and technologies like: SwiftUI, Core ML, Create ML, Core Image, Vision, SpriteKit, AVFoundation

The User Interface:

The user interface of the playground is developed in SwiftUI which enabled things like smooth animations when repositioning & resizing shapes, adjusting other parameters with the custom designed sliders and more in an elegant way. On the top of that, the intro animations are done using SpriteKit and thanks to the SwiftUI again which helps to enable seamless transitions from SpriteKit scenes to SwiftUI view.

Evaluating the created object:

When the user clicks on the 'Evaluate' button, an image of Canvas is captured. The Core Image framework helps preprocess the image by applying filters on it to remove any colors that might get captured due to selection of shapes.

The processed image is passed to a custom Core ML image classifier which is trained using Create ML with a special dataset. The output from the model will be a category in which the image is classified and the confidence of same. As per the confidence level of the label predicted, the output is presented to the user indicating whether their drawing is okay, along with other information like which and how many shapes are used to make the said object.

The Special Dataset:

For the dataset, I have developed an equivalent playground of this; It is slightly modified from this playground, where there's a button which instead of passing the image captured of canvas to the model, it gets saved locally as an image file. I have prepared a dataset of 105 images like this to train the model which can classify the images into the following categories: Home, Car, Camera, Cell Phone, Computer and Seesaw. (The dataset will be available publicly soon)

As of now, the Core ML model is trained with a limited number of images and a small dataset. Though it performs quite well, it can be improved even more with a larger dataset covering even more styles of structures for each category. Moreover, the dataset can also be expanded by introducing even more objects which can be easily formed using these three shapes only. Other than improving underlying machine learning model of evaluation, it can have even more features, like creating colorful objects to have an even more enjoyable experience, especially for kids. Unfortunately, I don't own an iPad yet, but I believe that this interface have a great potential to deliver an amazing experience equally on a touch first interface as in mouse and trackpad interface.

The Game of Shapes

Watch BlockDraw in Action

How it works?

Futurework