User Makeblock Computer Vision Extension for App Inventor

MIT App Inventor is a drag-and-drop block-based editor which can make Android Apps. Makeblock provides a simple computer vision plugin for color detection, face detection, and feature analysis (using online API).

Please also consider our mBot extension which allows you to control mBot robots.

Add the mBot extension to App Inventor

You need an App Inventor website that supports extensions. One example is Login and start a new project.


In the “Extension” group of the left panel, click “Import Extension” link, then select “URL”. Paste the following link to the text box:

Then click “import”.

Color Detection

Here is a sample program for color detection:

Screen Shot 2016-06-03 at 10.15.10 AM

The common process of color detection is: “Open the camera (at the first time)” -> “Take a picture” -> Analyze Color of the Picture” -> “Get RGB Values” -> “Take a picture again”.

When “Button 1” is clicked, the program opens the camera, then take a picture immediately.

Since taking the picture needs some time, the following steps are put into “After Picture Taken” blocks. In that block, “Analyze Color of the Picture” is firstly called  – this is needed before getting any color values. “Analyze Color of the Picture” calculates the mean color of the whole image, while you can get the color around a certain point using this command:

Screen Shot 2016-06-03 at 10.59.51 AM

Here the range of x, y is between 0-1. That means (0,0) is the left top corner of the image; (0.5, 0.5) is the image center, and (1,1) is the bottom right corner.

After analyzing the picture, I displayed the red, green, blue values of the picture to three text labels.

After all that, I took a picture again, which means “After Picture Taken” blocks will be called over and over.

Face Detection

Here is a sample program:

Screen Shot 2016-06-03 at 10.15.37 AM

When Button1 is clicked, I open the camera and start face detection. This is needed for all operations involving face detection.

“After face detection” block is called every time there’s a result about face detection. In this block, you can get number of faces using “Number of Faces” block; you may get the center and size of the faces detected.

Topic Analysis using Microsoft Cognitive Service

This plugin uses Microsoft’s Cognitive Service for topic analysis. Before using any of the following functions, you need to login and register an API account in Microsoft’s website:

After logged in, click “Request new trials”:

Screen_Shot_2016-06-03_at_11_18_29_AM 2

Then select “Computer Vision – Preview” to add computer vision API to your subscription list.


After that, you can check your API key by clicking “show” link. Copy that key, because it is needed in the following steps.


Here is a sample program of topic analysis:

Screen Shot 2016-06-03 at 10.30.40 AM

First, when Button1 is clicked, open the camera and Prepare computer vision with the API key mentioned before.

After Button2 (take the picture and analysis) is clicked, I take a picture. After the picture is taken, I use “Submit Picture for Computer Vision” to submit the picture to the Microsoft server.

When the result is returned from the server, “After Computer Vision Result” block will be called. Then I can get the following information:

  • DescriptionFromCV: a description in a line of text, such as “a man sitting in front of a chair”
  • Foreground/BackgroundColorNameFromCV: the name of the dominant color of the picture foreground/background. For example “blue”.
  • TagsFromCV: a list of tags describing the picture, such as “person”, “cat”, “desk”.

27 thoughts on “User Makeblock Computer Vision Extension for App Inventor”

  1. Perfect !
    I really appreciate this kind invitation to App Inventor Extensions in the field of computer vision.
    If you could show me where the picture taken by this program is stored in, I would be very happy.

  2. Thanks for this extension . Can you add a block that give the picture taken as its result? So that we could display it on screen.

    1. The picture taken is stored in the memory. I wonder how you’re going to use it, since no component in App Inventor allows loading an in-memory bitmap.

      App Inventor does not allow me to add display components in the current stage. Otherwise there could be camera previews and other fun stuff.

  3. This is the best step by step tutorial on the entire Makeblock site. Very impressive. I’d love to see this level of detail on some of the other pages.

  4. Hello everyone,
    I have run on the first simple tutorial of this page
    but it does not open the camera of my smartphone
    (Sony Xperia m Android 5.1.1)
    I have this error

    Runtime Error
    Attempt to invoke virtual method
    ‘android.hardware.Camera$Parameters()’ on a null object reference

    how can I solve?


  5. Question: I followed this for Face Detection, and when I hold the phone in front of me textforlabel1 just goes to 0. Shouldn’t something else be happening? Id like to use this as authentication when a user logs in they have to use Facial Recognition.

  6. Perfect, remains just the possibility to use front cam to combine between facial animation (canvas) and your computer vision.
    Thank you a lot and please let us know when you have news.

    1. Hi

      the “Open Camera” block itself will open the camera, although you can’t see a preview in the screen;
      Due to the limitation of App Inventor API, we can’t add a preview screen in the app. So sad…


    1. There’s no custom vision at this stage, and there’s no plan for the computer vision extension. Sorry to tell you that.

      But Makeblock, works with Microsoft, is always thinking about bringing AI to the classroom. So there could be such a move in other ways.

  7. Hi. I’m using this extension very well. But from about three days ago it suddenly show error. (Reading json error) do you know how to fix it? I’m in hurry. I need your help.

    1. Hi,

      That’s too bad. Perhaps something happens with Microsoft’s server – maybe they are changing the API or authentication method. I’ll tell the engineers to look at it, but it will take time.

      1. I’m so appreciated for your kind response. I’m a teacher. I use m-bot for SW education. And I use appinventor2. Your work is very helpful for teachers like me. I made a app for teaching with your extension. Without your extension, I couldn’t think about making the app. I will be waiting for your engineers help. Thank you very much.

  8. I followed the sample program u gave but even the camera isn’t opening on clicking the button 1..pls reply asap..

  9. hello ,
    i want to build an app that can detect rubik’s cube current color (cube status) ,
    than draw the detected color to a canvas ,
    your getPixelRGB can definitely do this,
    the only problem is that i don’t know where is my camera pixel 0,0 is located ,
    since you cant show a preview window how can you debug such an app ???

  10. Great extension !
    But I have an issue with “Analyze Point Color of the Picture”, it return abnormal Red (too low) and Green (too high) values…

    1. I found the error in the
      protected void getRGBValueOfImage(float px, float py)

      if (valuesCount > 0) {
      redValue /= valuesCount;
      blueValue /= valuesCount;
      redValue /= valuesCount;

      if (valuesCount > 0L) {
      redValue /= valuesCount;
      greenValue /= valuesCount;
      blueValue /= valuesCount;

  11. Hi

    I am really and sincerely impressed by the work that has been done. And the possibilities that it offers

    I have a question: how can we save the picture that has just been taken ? In all proposed methods and loops , I have not seen kind of “get picture” that could allow me to save the taken picture. After (for example) the face detection process is ended

    I would really appreciate your help.

  12. Hi
    I Try this code but it does not open the camera. The first project face detection is working a little bit: It shows how many faces are in the view. But I see no picture from the cam.
    The second project the same problem: no picture and I receive from Microsoft “I don’t know” and a json error 🙁
    I tested it with three smartphones and different Androids. App Inventor 2 Version 2.46
    Any hint?

Leave a Reply

Your email address will not be published. Required fields are marked *