NUWA Robot SDK

ver 1.1.2

How to start

Feature list:

System Requirement

Kebbi-air is based on Android P. You can use Android Studio or other IDE for android developing. Please refer to Android Studio for more information.

How to use

Get Nuwa SDK aar and import into your android studio project / or unity IDE.

Nuwa Robot SDK supports the following development environment.

Vendor could refer to "Example App" or "Tutorial App" to understand the API usage.

How to install APK

User can use ADB(android debug bridge) directly to install apk. (in Kebbi-Air)

After adb connected, developer can use adb install command to install apk.

Initiation

Before using Robot SDK, App needs to to create a single instance of Nuwa Robot API and do initiation once first.

    // ID name: please naming your own client id, 
    // Once mRobot is created, app will receive a callback of onWikiServiceStart()
    IClientId id = new IClientId("your_app_package_name");
    NuwaRobotAPI mRobot = new NuwaRobotAPI(android.content.Context, id);
        
    // release Nuwa SDK resource while App closed.(or Activity in pause/destroy state)
    mRobot.release();

Register callback

Due to all functions of Nuwa Robot SDK are async(Based on AIDL) design, App needs to register a callback to receive all kinds of notifications and results from Robot.

    mRobot.registerRobotEventListener(new RobotEventListener() {
        @Override
        public void onWikiServiceStart() {
            // Once mRobot is created, you will receive a callback of onWikiServiceStart()
        }
        @Override
        public void onTouchEvent(int type, int touch) {
            
        }
        .......
    });
    
    mRobot.registerVoiceEventListener(new VoiceEventListener() {
        @Override
        public void onTTSComplete(boolean isError) {
            // TODO Auto-generated method stub
            
        }
        .......
    });

You could also use helper class to overwrite events that App needs only.

    mRobot.registerRobotEventListener(new RobotEventCallback() {
        @Override
        public void onTouchEvent(int type, int touch) {
            
        }
    });
    
    mRobot.registerVoiceEventListener(new VoiceEventCallback() {
        @Override
        public void onTTSComplete(boolean isError) {
            
        }
    });

Check Nuwa Robot SDK engine is ready before app calling any Robot SDK API

Once mRobot is created, you will receive a callback of onWikiServiceStart()

NOTICE : please call api after onWikiServiceStart.

    mRobot.registerRobotEventListener(new RobotEventListener() {
        @Override
        public void onWikiServiceStart() {
            // Once mRobot is created, you will receive a callback of onWikiServiceStart()
        }
        .......
    });
    
    // or check state by
    boolean isReady = mRobot.isKiWiServiceReady();

Nuwa Motion player

Nuwa motion file is nuwa's private Robot motion control format. Which is composed of "MP4(Face)", "Motor control", "Timeline control", "LED control", etc. You can just play a motion, and Robot will do a serials of pre-defined actions. PS: Some motions only include "body movements" without "MP4(Face)"

    /*
        will get callbacks of onStartOfMotionPlay onStartOfMotionPlay(String motion)
        while finish plaing, will get callbacks of 
        onStopOfMotionPlay(String motion)
        onCompleteOfMotionPlay(String motion)
        
        // if there is an error, App will receive the error callback
        public void onErrorOfMotionPlay(int errorcode);
    */
    // use default NUWA motion asset path
    mRobot.motionPlay("001_J1_Good", true/false);
    
    // give a specefic motion asset path (internal use only)
    mRobot.motionPlay("001_J1_Good", true/false, "/sdcard/download/001/");
    //NOTICE:Please must call motionStop(true), if your auto_fadein is true.
    
    // will get callback of onStopOfMotionPlay(String motion)
    mRobot.stop(true/false);
    
    // preload motion, and then playback
    // callback to onPrepareMotion(boolean isError, String motion, float duration)
    mRobot.motionPrepare("001_J1_Good")
    
    // pause, resume and seek
    mRobot.motionPause();
    float pos = mRobot.motionCurrentPosition();
    mRobot.motionResume();
    mRobot.motionSeek(1.5f); // seek to 1.5 sec

Overlap window control

While playing a motion, there is a overlap window on the top of screen, you can do show or hide function with the window.

    mRobot.showWindow(false);
    mRobot.hideWindow(false);

Voice wakeup

You can say "Hello Kebbi" to get a wakeup event callback in your Appication context ONCE, and robot system behavior NOT response on this time.

        
    mRobot.startWakeUp(true);
    
    // When Robot hear "Hello Kebbi"，Kiwi agent will receive a callback onWakeup() of VoiceEventListener once
    onWakeup(boolean isError, String score) {
        // score: a json string given by voice engine
        // isError: if engine works normally
        // Ex: {"eos":3870,"score":80,"bos":3270,"sst":"wakeup","id":0}
        // score: confidence value
    }
    
    // stop listen to wakeup
    mRobot.stopListen();

Local TTS

Robot can speak out a sentance by a giving string

    mRobot.startTTS("Nice to meet you");
    
    // you can cancel speaking at any time
    mRobot.stopTTS();
    
    // receive callback onTTSComplete(boolean isError) of VoiceEventListener
    onTTSComplete(boolean isError) {
    }

Local ASR

ASR engine use SimpleGrammarData to describe the list of keyword, App needs to create Grammar first. (Command table)

Example

    //list listen syntax
    ArrayList<String> cmd = new ArrayList<>();
    cmd.add("I want to listen music");
    cmd.add("play pop music");
    cmd.add("listen love song");
    //Make Grammar Object by SimpleGrammarData class
    SimpleGrammarData mGrammarData = new SimpleGrammarData("TutorialTest");
    for (String string : cmd) {
      mGrammarData.addSlot(string);
    }
    //update config
    mGrammarData.updateBody();
    //Regist local ASR syntax
    mRobot.createGrammar(mGrammarData.grammar, mGrammarData.body);
        
    // stop ASR operation
    mRobot.stopListen();

VoiceEventListener Callback

    // receive a GrammarState callback of VoiceEventListener
    public void onGrammarState(boolean b, String s) {
      // now you can call startLocalCommand() API
      mRobot.startLocalCommand();
    }
    
    // receive a Understand callback of VoiceEventListener
    public void onMixUnderstandComplete(boolean b, ResultType resultType, String s) {
      //Get ASR result string here
      String result_string = VoiceResultJsonParser.parseVoiceResult(s);
      
    }

Cloud Speech To Text and Local ASR mix operation

While doing ASR Mix mode, engine will receive results from local and cloud, engine will only return one of both. Rules are:

  1. if engine get local result, it will only return local result
  2. if engine get no local result, it will return ASR result if Internet is available

PS: Cloud ASR and local ASR result are json format string.

Example

    //list listen syntax
    ArrayList<String> cmd = new ArrayList<>();
    cmd.add("I want to listen music");
    cmd.add("play pop music");
    cmd.add("listen love song");
    //Make Grammar Object by SimpleGrammarData class
    SimpleGrammarData mGrammarData = new SimpleGrammarData("TutorialTest");
    for (String string : cmd) {
      mGrammarData.addSlot(string);
    }
    //update config
    mGrammarData.updateBody();
    //Regist local ASR syntax
    mRobot.createGrammar(mGrammarData.grammar, mGrammarData.body);
    
    
    // stop ASR operation
    mRobot.stopListen();

VoiceEventListener Callback

    // receive a callback of VoiceEventListener
    onGrammarState(boolean isError, String info) {
    // now you can call startMixUnderstand() API
    // do mix mode ASR
      mRobot.startMixUnderstand();
    }
    
    // get local
    onMixUnderstandComplete(boolean isError, ResultType type, String json) {
      //Get ASR result string
      String result_string = VoiceResultJsonParser.parseVoiceResult(json);
      //get ASR type
      if (type == ResultType.LOCAL_COMMAND) {
        //do something
      }
    }

Sensor event

Robot provides Touch and PIR and Dropsensor events. While you need it, you can request it. While you don't need it, you can stop requesting it.

    public static final int SENSOR_NONE = 0x00;             //000000
    public static final int SENSOR_TOUCH = 0x01;            //000001
    public static final int SENSOR_PIR = 0x02;              //000010
    public static final int SENSOR_DROP = 0x04;             //000100
    public static final int SENSOR_SYSTEM_ERROR = 0x08;     //001000
    
    // request touch sensor event
    mRobot.requestSensor(SENSOR_TOUCH);
    
    // or request touch, PIR event and SENSOR_XXX
    mRobot.requestSensor(NuwaRobotAPI.SENSOR_TOUCH | NuwaRobotAPI.ENSOR_PIR | NuwaRobotAPI.SENSOR_XXX);
    
    // get raw touch results of RobotEventListener
    // type: head: 1, chest: 2, right hand: 3, left hand: 4, left face: 5,right face: 6.
    // touch: touched: 1, untouched: 0
    onTouchEvent(int type, int touch) {
    }
    
    // type: head: 1, chest: 2, right hand: 3, left hand: 4, left face: 5,right face: 6.
    onTap(int type) {
    }
    
    // type: head: 1, chest: 2, right hand: 3, left hand: 4, left face: 5,right face: 6.
    onLongPress(int type) {
    }
    
    // get PIR results of RobotEventListener
    onPIREvent(int val) {
    }

    // stop all requested sensor event
    mRobot.stopSensor(NuwaRobotAPI.SENSOR_NONE);
    
    // stop multiple sensor events
    mRobot.stopSensor(NuwaRobotAPI.SENSOR_TOUCH | NuwaRobotAPI.SENSOR_XXX);

LED control

There are 4 parts of LED on Robot. API can contorl each them. (Head, Chest, Right hand, Left hand) Each LED part has 2 types of modles - "Breath mode" and "Light on mode" Before using it, you need to use API to turn on it first, and turn off it while unneeded. LED default controled by System. If the App wants to have different behavior, it can be disabled by disableSystemLED()
If your App needs to control Robot LED, App needs to call disableSystemLED() once，and App call enableSystemLED() while App is in onPause state.

    /*
    id:1 = Face LED
    id:2 = Chest LED
    id:3 = Left hand LED
    id:4 = Right hand LED
    onOff:  0 or 1
    brightness, Color-R, Color-G, Color-B: 0 ~ 255
    interval: 0 ~ 15 
    ratio: 0 ~ 15
    */
    
    // turn on LED
    mRobot.enableLed(1, 1);
    mRobot.enableLed(2, 1);
    mRobot.enableLed(3, 1);
    mRobot.enableLed(4, 1);
    
    // Set LED color
    mRobot.setLedColor(1, 255, 255, 255, 255);
    mRobot.setLedColor(2, 255, 255, 0, 0);
    mRobot.setLedColor(3, 255, 166, 255, 5);
    mRobot.setLedColor(4, 255, 66, 66, 66);
    
    // Switch to "Breath mode"
    mRobot.enableLedBreath(1, 2, 9);
    
    // turn off LED
    mRobot.enableLed(1, 0);
    mRobot.enableLed(2, 0);
    mRobot.enableLed(3, 0);
    mRobot.enableLed(4, 0);

Motor control

Mibo Robot has 10 motors, use the API, you can control each of them

Alt text

Motor angle range table

ID 1 2 3 4 5 6 7 8 9 10
Max 20 40 5 70 100 0 5 70 100 0
Min -20 -40 -85 -200 -3 -80 -85 -200 -3 -80 
  // control neck_y motor to 20 degree in 40 Degree/sec speed 
  mRobot.ctlMotor(1, 0, 20, 40f);

Movement control

To control Robot to forward, backwards, turns, stop

    // go forward
    mRobot.forwardInAccelerationEx();
    // go back
    mRobot.backInAccelerationEx();
    // stop
    mRobot.stopInAccelerationEx();

Safe mode

Make Robot could stand in place at any case. Robot will auto enable "lock wheel mode" while AC is plugged. App could call "unlockWheel()" to disable the safe mode.

Handle Robot Drop event

When Robot drop happening, it will deliver error message and App could receive the error message via RobotEventListener callback

    // request Robot drop event first
    mRobot.requestSensor(NuwaRobotAPI.SENSOR_DROP);
    
    // handle it 
    public void onDropSensorEvent(int value) {
        // value: 1 : drop
                  0 : normal
        
        // get the amount of drop IR sensor 
        int val = mRobot.getDropSensorOfNumber();
    }

    // release sensor event
    mRobot.stopSensor(NuwaRobotAPI.SENSOR_NONE);

Handle Robot Service Recovery

When Robot service(Robot SDK) happens unexpected exception, it will restart automaticaly. App could handle the scenario by following callback of RobotEventListener.

    public void onWikiServiceStart() {
        //1. Robot SDK is ready to use now.
        //2. Robot SDK restart successfully, and it's ready to use now.
        Log.d(TAG, "onWikiServiceStart");
    }

    @Override
    public void onWikiServiceCrash() {
        // When Robot service(Robot SDK) happens unexpected exception, it will shutdown itself.
        Log.d(TAG, "onWikiServiceCrash");
    }

    @Override
    public void onWikiServiceRecovery() {
        // Robot service(Robot SDK) begins to restart itself.
        // When it's ready, it will send "onWikiServiceStart" event to App again.
        Log.d(TAG, "onWikiServiceRecovery");
    }

How to shown entrypoint on Robot Menu

Robot-air menu base on Launcher, please implement standard launcher icon. AndroidManifest.xml

    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:roundIcon="@mipmap/ic_launcher_round"
        android:supportsRtl="true"
        android:theme="@style/AppTheme">
        <activity
            android:name=".MainActivity"
            android:label="@string/app_name"
            android:theme="@style/AppTheme.NoActionBar">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>

Android Developer reference link https://developer.android.com/guide/topics/manifest/manifest-intro#iconlabel

Launch Developer App via Voice command

In Original Android design, App is launched by tapping icon in Launcher. In NUWA Robot we allow use "Voice command" to launch an activity or broadcast an intent to a registered receiver.

*Developer can setup several "Voice Command" separate by ",".

NOTICE: "Voice Command" have to be 100% matched.

    Ex: in your manifest, add the "intent-filter" of "com.nuwarobotics.api.action.VOICE_COMMAND" in "activity" to identify the activity wants to be launched by Robot Launcher while Robot hears voice commands like [Play competition game,Play student competition game,I want play student game,student competition].
    
    <activity
        android:name=".YouAppClass"
        android:exported="true"
        android:label="App Name"
        android:screenOrientation="landscape" >

        <intent-filter>
            <action android:name="com.nuwarobotics.api.action.VOICE_COMMAND" />
        </intent-filter>

        <meta-data
            android:name="com.nuwarobotics.api.action.VOICE_COMMAND"
            android:value="Play competition game,Play student competition game,I want play student game,student competition" />
    </activity>
    
    Ex: in your manifest, add the "intent-filter" of "com.nuwarobotics.api.action.VOICE_COMMAND" in "receiver" to identify the receiver wants to be launched by Robot Launcher while Robot hears voice commands like [I want to watch TV,I want to sleep,I want to turn on air conditioner].
    <receiver android:name=".VoiceCommandListener" >
        <intent-filter>
            <action android:name="com.nuwarobotics.api.action.VOICE_COMMAND" />
        </intent-filter>

        <meta-data
            android:name="com.nuwarobotics.api.action.VOICE_COMMAND"
            android:value="I want to watch TV,I want to sleep,I want to turn on air conditioner" />
    </receiver> 
    
    // in VoiceCommandListener.java
    public class VoiceCommandListener extends BroadcastReceiver{
        @Override
        public void onReceive(Context context, Intent intent) {
        	Log.d(TAG, "action:" + intent.getAction());
    		if (intent.getAction().equals("com.nuwarobotics.api.action.VOICE_COMMAND")) {
    			String cmd = intent.getStringExtra("cmd");
    			Log.d(TAG, "user speak: " + cmd);
    		}
    	}
    }

Disable Support Always Wakeup

Kebbi-air support "Always wakeup" anywhere. (Kebbi and Danny only support wakeup on Face) We allow App declare on Application or Activity scope.

Declare on AndroidManifest.xml
 <meta-data android:name="disableAlwaysWakeup" android:value="true" />

Application Scope Example

    <application
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:theme="@style/AppTheme">
        
        <meta-data android:name="disableAlwaysWakeup" android:value="true" />
        
    </application>

Activity Scope Example

    <activity
        android:name=".YouAppClass"
        android:exported="true"
        android:label="App Name"
        android:screenOrientation="landscape" >

        <meta-data android:name="disableAlwaysWakeup" android:value="true" />
        
    </activity>

Q & A

Q1: Why it dodn't work while App calls "mRobot.motionPlay("001_J1_Good", true/false)"?
Ans:

  1. check the callback of "public void onErrorOfMotionPlay(int errorcode) to know the error info"
  2. make sure the motion name is valid
  3. check the motion list to get the correct motion name

Q2: Why App can not control LED?
Ans:

  1. App needs to call disableSystemLED() once first. Ex: Do it in onCreate() state.
  2. App needs to call enableSystemLED() while App closed. Ex: Do it in onPause() state.

Q3: Why App GUI widget(Ex: Button) can not receive any touch event?
Ans:

  1. There is an overlap window on your app, App call "hideWindow(false)" to close the window.

Q4: Why App GUI widget(Ex: button) can not receive any touch event after playing a motion?
Ans:

  1. There is an overlap window on your app, App call "hideWindow(false)" to close the window in the callback of onCompleteOfMotionPlay().

Q5: Does local ASR and cloud ASR support English?
Ans:

  1. Support "Japanese" and "English".

Q6: Does local TTS support English?
Ans:

  1. TTS supports "Japanese" and "English"

Q7: Why App can not receive Robot touch event?
Ans:

  1. App calls "mRobot.requestSensor(SENSOR_TOUCH)" once.
  2. Handle the following callback
    mRobot.registerRobotEventListener(new RobotEventListener() {
        @Override
        public void onTouchEvent(int type, int touch) {
            // type: head: 1, chest: 2, right hand: 3, left hand: 4, left face: 5,right face: 6.
            // touch: touched: 1, untouched: 0
        }
    });
  1. App calls "mRobot.stopSensor(NuwaRobotAPI.SENSOR_NONE)" while App doesn't want to receive the touch events anymore.