User Makeblock Computer Vision Extension for App Inventor

MIT App Inventor is a drag-and-drop block-based editor which can make Android Apps. Makeblock provides a simple computer vision plugin for color detection, face detection, and feature analysis (using online API).

Please also consider our mBot extension which allows you to control mBot robots.

Add the mBot extension to App Inventor

You need an App Inventor website that supports extensions. One example is http://extension-test.appinventor.mit.edu/. Login and start a new project.

app-inventor-1

In the “Extension” group of the left panel, click “Import Extension” link, then select “URL”. Paste the following link to the text box:

http://appinventor.makeblock.com/com.makeblock.appinventor.CamVision.aix

Then click “import”.

Color Detection

Here is a sample program for color detection:

Screen Shot 2016-06-03 at 10.15.10 AM

The common process of color detection is: “Open the camera (at the first time)” -> “Take a picture” -> Analyze Color of the Picture” -> “Get RGB Values” -> “Take a picture again”.

When “Button 1” is clicked, the program opens the camera, then take a picture immediately.

Since taking the picture needs some time, the following steps are put into “After Picture Taken” blocks. In that block, “Analyze Color of the Picture” is firstly called  – this is needed before getting any color values. “Analyze Color of the Picture” calculates the mean color of the whole image, while you can get the color around a certain point using this command:

Screen Shot 2016-06-03 at 10.59.51 AM

Here the range of x, y is between 0-1. That means (0,0) is the left top corner of the image; (0.5, 0.5) is the image center, and (1,1) is the bottom right corner.

After analyzing the picture, I displayed the red, green, blue values of the picture to three text labels.

After all that, I took a picture again, which means “After Picture Taken” blocks will be called over and over.

Face Detection

Here is a sample program:

Screen Shot 2016-06-03 at 10.15.37 AM

When Button1 is clicked, I open the camera and start face detection. This is needed for all operations involving face detection.

“After face detection” block is called every time there’s a result about face detection. In this block, you can get number of faces using “Number of Faces” block; you may get the center and size of the faces detected.

Topic Analysis using Microsoft Cognitive Service

This plugin uses Microsoft’s Cognitive Service for topic analysis. Before using any of the following functions, you need to login and register an API account in Microsoft’s website:

https://www.microsoft.com/cognitive-services/en-US/subscriptions

After logged in, click “Request new trials”:

Screen_Shot_2016-06-03_at_11_18_29_AM 2

Then select “Computer Vision – Preview” to add computer vision API to your subscription list.

Screen_Shot_2016-06-03_at_11_19_33_AM

After that, you can check your API key by clicking “show” link. Copy that key, because it is needed in the following steps.

Screen_Shot_2016-06-03_at_11_18_29_AM

Here is a sample program of topic analysis:

Screen Shot 2016-06-03 at 10.30.40 AM

First, when Button1 is clicked, open the camera and Prepare computer vision with the API key mentioned before.

After Button2 (take the picture and analysis) is clicked, I take a picture. After the picture is taken, I use “Submit Picture for Computer Vision” to submit the picture to the Microsoft server.

When the result is returned from the server, “After Computer Vision Result” block will be called. Then I can get the following information:

  • DescriptionFromCV: a description in a line of text, such as “a man sitting in front of a chair”
  • Foreground/BackgroundColorNameFromCV: the name of the dominant color of the picture foreground/background. For example “blue”.
  • TagsFromCV: a list of tags describing the picture, such as “person”, “cat”, “desk”.

14 thoughts on “User Makeblock Computer Vision Extension for App Inventor”

  1. Perfect !
    I really appreciate this kind invitation to App Inventor Extensions in the field of computer vision.
    If you could show me where the picture taken by this program is stored in, I would be very happy.

  2. Thanks for this extension . Can you add a block that give the picture taken as its result? So that we could display it on screen.

    1. The picture taken is stored in the memory. I wonder how you’re going to use it, since no component in App Inventor allows loading an in-memory bitmap.

      App Inventor does not allow me to add display components in the current stage. Otherwise there could be camera previews and other fun stuff.

  3. This is the best step by step tutorial on the entire Makeblock site. Very impressive. I’d love to see this level of detail on some of the other pages.

  4. Hello everyone,
    I have run on the first simple tutorial of this page
    but it does not open the camera of my smartphone
    (Sony Xperia m Android 5.1.1)
    I have this error

    Runtime Error
    Attempt to invoke virtual method
    ‘android.hardware.Camera$Parameters
    ‘android.hardware.Camera$Parameters()’ on a null object reference

    how can I solve?

    Regards

  5. Question: I followed this for Face Detection, and when I hold the phone in front of me textforlabel1 just goes to 0. Shouldn’t something else be happening? Id like to use this as authentication when a user logs in they have to use Facial Recognition.

  6. Perfect, remains just the possibility to use front cam to combine between facial animation (canvas) and your computer vision.
    Thank you a lot and please let us know when you have news.

    1. Hi

      the “Open Camera” block itself will open the camera, although you can’t see a preview in the screen;
      Due to the limitation of App Inventor API, we can’t add a preview screen in the app. So sad…

      Yu

Leave a Reply

Your email address will not be published. Required fields are marked *