Advanced Sample App: Mad Hatter

Audience

This article is for Android programmers who are already familiar with the use of the Camera.
(http://developer.android.com/reference/android/hardware/Camera.html).

Introduction

In this article the subject of face detection and drawing will be addressed. New methods of automatic face detection in Android will be discussed, as well as ideas on what to do, once the faces have been detected. For the purpose of this document, the author assumes that the reader possesses basic programming skills and basic Android knowledge.

Getting started

In Android there are two ways of detecting faces: the Media.FaceDetector class, which has been around since API Level 1, and the new Camera.Face class, that has been brought in with the API 14 update.
Both these methods are quite easy to use, but they require some preparation.
The Media approach uses the Media.FaceDetector class on static bitmaps, which allows facial detection on existing pictures, whereas the Camera approach can work on the fly.

App development

Before coding

This article will explain the Camera.Face method of using face detection. This method was implemented in Android 4.0 and has been used in this system for security measures – the user can now unlock their phone just by looking at it. This method is much quicker than the previously mentioned one and can work on the fly with a preview picture from the camera.

Camera preview

The first thing needed is a simple preview class. It should be a surface view, that responds to callbacks and starts facial recognition.

public CameraPreview(Context context, Camera camera) {
	super(context);
	mCamera = camera;
	mHolder = getHolder();
	mHolder.addCallback(this);
}

@Override
public final void surfaceCreated(SurfaceHolder holder) {
	try {
		mCamera.setPreviewDisplay(holder);
		mCamera.startPreview();
		MadHatter.resetRotation();
		mCamera.startFaceDetection();
	} catch (IOException e) {
		Log.d(TAG, "Error setting camera preview: " + e.getMessage());
	} catch (RuntimeException e) {
		Log.d(TAG, "Error setting camera preview: " + e.getMessage());
	}
}

The constructor and surfaceCreated method in the camera preview class.

Face detection listener

The goal of this app is to put a top hat on each head found in the preview. Once a face detection listener has been created and attached to the activity, an array of faces will be received each time these have been detected. Another important thing is to know when no faces have been detected. This is important, since old hats have to be removed from the preview. This is done in two places – first each time the onFaceDetection listener is fired, the layout is cleared of all the faces. This is launched even when no faces have been detected, since the listener listens on, but returns an empty Face array. The second place is in the rotation listener – when the device has been rotated, all hats are destroyed.

@Override
public final void onFaceDetection(Face[] faces, Camera camera) {
	clearHatLayout();
	mFacesList.clear();

	if (faces.length > 0) {
		for (Face face : faces) {
			if (face != null) {
				mFacesList.add(face);
			}
		}
		hattenize();
	}
}

What happens when faces are detected.

Each time the listener receives an array of faces it fires up the hattenize() method, which creates a new HatView, with the current app context, the list of faces, the current bitmap to be drawn on top of them, and the height and width of the drawing space and adding the HatView to the layout.

public final void hattenize() {
mHatView = new HatView(mContext, mFacesList, mWidth, mHeight, mHatBitmap);
mHatLayout.addView(mHatView);}

HatView

The HatView is a class extending the View, its basic function is to draw the hats upon the faces received from the listener. It draws them based on the calculated coordinates turned to the rotation of the device. The important things here are these coordinates. Even though the Camera.Face class provides fields such as Point leftEye, Point mouth, Point rightEye and Rect rect, the first three do not work on all devices, and if not supported will return a null value.

If they are supported, the coordinates of the eyes permit calculating the angle of the head’s tilt and an accurate width of the face. If not, the only thing left is a rectangle that gives only roughly where the face is situated.

One more important thing to take into account, is that the rect field in the Face class gives the position not in pixels on the screen, but in a coordinate system, where the top left corner is represented by (-1000, -1000) and the bottom right by (1000, 1000). To get a faces coordinates as on the preview, it is necessary to normalize the values:

divWidth = (double) this.mWidth / MadHatter.COORDINATE_NORMALIZE;
divHeight = (double) this.mHeight / MadHatter.COORDINATE_NORMALIZE;

mRectanF.set((float) ((face.rect.left + MadHatter.ZERO_NORMALIZE) * mDivWidth),
	(float) ((face.rect.top + MadHatter.ZERO_NORMALIZE) * mDivHeight - face.rect.height() * mDivHeight), 
	(float) ((face.rect.right + MadHatter.ZERO_NORMALIZE) * mDivWidth),
	(float) ((face.rect.bottom + MadHatter.ZERO_NORMALIZE) * mDivHeight - face.rect.height() * mDivHeight));

mRectan.set(0, 0, mHatBitmap.getWidth(), mHatBitmap.getHeight());
canvas.drawBitmap(mHatBitmap, mRectan, mRectanF, mPaint);

HatView onDraw method

where the COORDINATE_NORMALIZE constant is 2000 and ZERO_NORMALIZE is 1000.
This also sets the size of the hat, as it is calculated based on the size of the face.
These coordinates will give the position of the face, however they only apply to the horizontal view of the app so the coordinates’ orientation needs to be adjusted.

Orientation

For the rotation of the device an OrientationEventListener will be used. Based on its readings of the OrientationEventListner the bitmaps and buttons will be rotated.

private void setOrientationListener() {
mOrientListener = new OrientationEventListener(getApplicationContext()) {
    @Override
    public void onOrientationChanged(int orientation) {
        if (mUnRotatedBitmap == null) {
            return;
        }
        mPreviousRotation = sRotate;

        if ((orientation >= (DEGREES_360 - DEGREES_45) && orientation < DEGREES_360) || 
		(orientation >= 0 && orientation < DEGREES_45)) {
            if (sRotate != Surface.ROTATION_0) {
            setRotation(Surface.ROTATION_0);
            rotateHat(DEGREES_180);
                rotateButtons(switchOrientation(Surface.ROTATION_90, Surface.ROTATION_270));}

{ ... }
	}
};
mOrientListener.enable();
}

Setting the orientation listener

The orientation of the device influences the LayoutParam. These parameters will set the size of the preview and the whole layout, so that it maintains a proper image format, without stretching it.

private void setPreviewAndLayouts(Camera camera) {
mPreview = new CameraPreview(this, camera);
float width;
float height;
WindowManager myWindowManager = (WindowManager) getApplicationContext()
		.getSystemService(Context.WINDOW_SERVICE);
if (myWindowManager != null) {
	Display display = myWindowManager.getDefaultDisplay();
	/*
	 * checking for the rotation and adjusting the view accordingly
	 */
	setContentView(R.layout.main);
	width = display.getHeight();
	height = (width * IMAGE_FORMAT);
	mParams = new LayoutParams((int) height, (int) width);
	mCamera.setDisplayOrientation(0);

	mPreview.setLayoutParams(mParams);
	mHatLayout = (RelativeLayout) findViewById(R.id.camera_preview);
	mHatLayout.setLayoutParams(mParams);
	mHatLayout.addView(mPreview);

	mFaceDetectionListener = new HatterFaceDetectionListener(getApplicationContext(), mHatLayout, 
	mHatBitmap);
	mFaceDetectionListener.setParams(mParams.width, mParams.height);
	mCamera.setFaceDetectionListener(mFaceDetectionListener);

    }
}

Setting previews, layouts and basic options

Each time the rotation gets changed, the setRotation method has to be called, so that other classes know which side is up. The rotation of the hat and the buttons also change, so that it is always shown in the upright position.
Since the app is set to run only in horizontal mode, the layout doesn’t change automatically and it has to change accordingly. Animations to turn the buttons have been implemented.

mRotateAnimation = new RotateAnimation((DEGREES_270 - (mPreviousRotation * DEGREES_90)) % 
DEGREES_360,
((DEGREES_270 - mPreviousRotation * DEGREES_90)) % DEGREES_360 + rotationDegrees,
Animation.RELATIVE_TO_SELF, 0.5f, Animation.RELATIVE_TO_SELF, 0.5f);

mRotateAnimation.setInterpolator(new LinearInterpolator());
mRotateAnimation.setDuration(ANIMATION_TIME);
mRotateAnimation.setFillEnabled(true);
mRotateAnimation.setFillAfter(true);

mHatButton.startAnimation(mRotateAnimation);
mFlipper.startAnimation(mRotateAnimation);

One of the button orientation animations

Saving and sharing

The next step is to save the picture.
Saving the contents of the preview is easy, as all that is needed is to call the takePicture() method of the camera, but that would only save what the camera sees, without the hats. To add those, the byte array received in the onPictureTaken method has to be edited. The byte array has to be decoded to a bitmap. Hat bitmaps will then be added to the canvas created on the background and then the resulting canvas needs to be coded to a byte array, so it can be saved to file.

private void drawHatsOnCanvas(Canvas canvas, Bitmap hatBitmap, double divWidth, double divHeight) {

Paint paint = new Paint();
RectF rectan = new RectF();
// Get the last list of faces
for (Face face : mFaceDetectionListener.getFacesList()) {

/*
 * normalizing the coordinates from (-1000, 1000) to (0, width) and
 * (0, height)
 */

rectan.set((float) ((face.rect.left + ZERO_NORMALIZE) * divWidth),
    (float) ((face.rect.top + ZERO_NORMALIZE) * divHeight - face.rect.height() * divHeight),
    (float) ((face.rect.right + ZERO_NORMALIZE) * divWidth),
    (float) ((face.rect.bottom + ZERO_NORMALIZE) * divHeight - face.rect.height() * divHeight));

canvas.drawBitmap(hatBitmap, new Rect(0, 0, hatBitmap.getWidth(), hatBitmap.getHeight()), rectan, paint);

Drawing bitmaps on a canvas

 private byte[] addHatsToBitmap(byte[] bitmapByteArray) {

    BitmapFactory.Options bfo = new BitmapFactory.Options();
    bfo.inMutable = true;
    // checking if not using too much memory, if yes, compress bitmap
    if (bitmapByteArray.length > MAX_BITMAP_SIZE) {
        bfo.inSampleSize = 2;
    }
    // decoding the byte array to get the picture captured by camera
    Bitmap bitmapWithoutHats;
    bitmapWithoutHats = BitmapFactory.decodeByteArray(bitmapByteArray, 0, bitmapByteArray.length, bfo);

    if (mUnRotatedBitmap != null) {
        Bitmap turnedBitmap = null;

        if (bitmapWithoutHats != null) {
            turnedBitmap = rotateBitmap(bitmapWithoutHats);
            bitmapWithoutHats.recycle();
            // setting a canvas, to which we'll be drawing the hats
            Canvas canvas = new Canvas(turnedBitmap);

            // setting the divisors for normalizing the coordinates
            double divWidth = (double) bitmapWithoutHats.getWidth() / COORDINATE_NORMALIZE;
            double divHeight = (double) bitmapWithoutHats.getHeight() / COORDINATE_NORMALIZE;
            drawHatsOnCanvas(canvas, mUnRotatedBitmap, divWidth, divHeight);
        }

        if (turnedBitmap != null) {
            ByteArrayOutputStream stream = new ByteArrayOutputStream();
            turnedBitmap.compress(Bitmap.CompressFormat.PNG, 100, stream);
            byte[] turnedBitmapByteArray = stream.toByteArray();
            try {
                stream.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
            return turnedBitmapByteArray;
        }
    }
    return bitmapByteArray;
}

Decoding byte array, drawing hats, coding back to byte array

It is important to remember the rotation of the screen, and rotating the picture, so it faces up. A “Share” button functionality have been added, so that the user can share the picture with the hat on his/her head.

private void sharePicture() {
    File file = new File(Environment.getExternalStorageDirectory(), TMP_FILENAME);

    OutputStream output = null;

    try {
        Bitmap bitmap = loadBitmapFromFile(mBitmapFile);
        output = new BufferedOutputStream(new FileOutputStream(file));
        if (bitmap != null) {
            bitmap.compress(Bitmap.CompressFormat.PNG, 100, output);
            Intent sendIntent = new Intent(Intent.ACTION_SEND);
            sendIntent.setType("image/png");
            sendIntent.putExtra(Intent.EXTRA_STREAM, Uri.fromFile(file));
            startActivity(Intent.createChooser(sendIntent, getString(R.string.share)));
        }
    } catch (FileNotFoundException e) {
        Toast.makeText(this, R.string.share_failed, Toast.LENGTH_SHORT).show();
    } catch (IOException e) {
        Toast.makeText(this, R.string.load_failed, Toast.LENGTH_SHORT).show();
        e.printStackTrace();
    } finally {
        try {
            if (output != null) {
                output.close();
            }
        } catch (IOException e) {
            Toast.makeText(this, e.getMessage(), Toast.LENGTH_SHORT).show();

        }
    }
}

Sharing functionality

Changing the hat bitmap

A button to change the hat bitmap has also been added. Three bitmaps are available, so after pressing the button its icon changes and so does the hat on the head. This is done via a ViewFlipper. Each button is added to the flipper, and after clicking one button, the next appears in its place, therefore a button has to be created for each hat.

blackButton.setOnClickListener(new OnClickListener() {
    @Override
    public void onClick(View v) {
        mHatBitmap = BitmapFactory.decodeResource(getResources(), R.drawable.hat3);
        mUnRotatedBitmap = BitmapFactory.decodeResource(getResources(), R.drawable.hat3);
        mFlipper.setDisplayedChild((mHatButtonState++ % NUMBER_OF_BITMAPS));
        resetRotation();
//The rotation is being reset, so that the hats have to be redrawn.

    }
});

One of the buttons created

private void setViewFlipper() {
    mFlipper = (ViewFlipper) findViewById(R.id.flipper);
    mSlideIn = AnimationUtils.loadAnimation(this, R.anim.slidein);
    mSlideOut = AnimationUtils.loadAnimation(this, R.anim.slideout);
    mFlipper.setInAnimation(mSlideIn);
    mFlipper.setOutAnimation(mSlideOut);
}

Setting the view flipper

<?xml version="1.0" encoding="utf-8"?>

<set xmlns:android="http://schemas.android.com/apk/res/android" 
	android:interpolator="@android:anim/decelerate_interpolator">
	 
    <translate android:fromXDelta="-100%" android:toXDelta="0%" android:duration="100" />
	
</set>

The “in” animation. “Out” animation is created in the same way

The sample code to show a camera preview, detect faces on it and draw a top hat over each one of them has been attached to this article.