Matrix Assignment and Cloning

In opencv, the assignment (“=”) operator does not copy a matrix. It only attaches a new label or alias to a matrix. In c/c++ when we write A = B, we expect A and B to be two different variables with same value. However, in OpenCV, such an assignment operation will actually say that A and B are invariably the same matrix. This will be illustrated in the following opencv snippet.

Program Code	Program Output
cv::Mat MyMat = cv::Mat::eye(3,3,CV_64FC1); cv::Mat AnotherMat = MyMat; AnotherMat.at(1,2) = 5.5; std::cout<<MyMat;

the clone() method. For example, if we change the 2nd line of this code by the following line, then “MyMat” will not be changed anymore when we’ll change the ”AnotherMat” matrix.

cv::Mat AnotherMat = MyMat.clone();

When we use the clone method, a deep copy of all the elements take place. So, for big matrices, the clone operation may take considerably longer time than the assignment operation. According to the language of OpenCV documentation, the matrix data is copied in clone operation whereas only the matrix header is copied in simple assignments. Header actually means some basic information about a matrix such as, the number of rows, columns, number of channels, data type etc. Sometimes, we might need to change the header. To do that, we can use the create() method. The following code will clarify its use.

cv::Mat MyMat = cv::Mat::eye(3,3,CV_64FC1);
MyMat.create(2,2,CV_64FC1);
MyMat.setTo(5);
std::cout<<MyMat;

The MyMat was a 3×3 double type single channel matrix. However, the create() method in line 2 has changed the header into a 2×2 double type single channel matrix. The setTo() method in line 3 is used to set all the elements of the matrix equal to to 5.0.

Accessing Sub-parts of Matrices

While working with matrices in OpenCV, we often need to access into a matrix partially. For instance, we sometimes need to access the rows, columns, or diagonal elements of a matrix. We often also need to concatenate two matrices (or vectors) horizontally or vertically. There is a set of functions in OpenCV (row, col, diag, operator(), rowrange, colrange, hconcat, vconcat, copyto) to support these operations. The following code illustrates the use of these functions.

cv::Mat MyMat = cv::Mat::eye(3,3,CV_64FC1);
std::cout<<MyMat<<std::endl;
cv::Mat oneRow = MyMat.row(1);
std::cout<<oneRow<<std::endl;
// Also play with col

oneRow.setTo(5.0);
std::cout<<oneRow<<std::endl;
std::cout<<MyMat<<std::endl;
cv::Mat twoRows = MyMat.rowRange(1,3).clone();
// Notice that it is not (1,2)
std::cout<<twoRows<<std::endl;
twoRows.copyTo(MyMat.rowRange(0,2));
std::cout<<MyMat<<std::endl;
std::cout<<oneRow<<std::endl;
// Also play with colRange

MyMat(cv::Range(1,3),cv::Range(0,2)).setTo(1.5);
std::cout<<MyMat<<std::endl;
std::cout<<MyMat.diag()<<std::endl;
MyMat.diag().setTo(3.33);
std::cout<<MyMat<<std::endl;

[1,0,0;
0,1,0;
0,0,1]
[0,1,0]
[5,5,5]
[1,0,0;
5,5,5;
0,0,1]
[5,5,5;
0,0,1]
[5,5,5;
0,0,1;
0,0,1]
[0,0,1]
[5,5,5;
1.5,1.5,1;
1.5,1.5,1]
[5;1.5;1]
[3.33,5,5;
1.5,3.33,1;
1.5,1.5,3.33]

We need to analyze this code very carefully. In lines 1 and 2, a 3×3 identity matrix has been created and displayed. In line 3, the matrix “oneRow” is defined for pointing to the second row of “MyMat”. Unlike MATLAB, the row method uses a 0-based indexing. In fact, 0-based indexing is consistently used in all the functions of OpenCV. Notice in line 3 that, the assignment operator is used without any cloning operation. As a result, if we change oneRow, this will be reflected in MyMat. This is shown in line 7-9 of the code. When we use the setTo command to set all the elements of oneRow to become 5.0, it also changes the 2nd row of MyMat (look at lines 6-8 of program output). This kind of operation should be handled very carefully because this behavior often cause confusions while working with OpenCV. Similar operations can also be done with col method, where the columns will be manipulated.

In line 10 of the program code, rows second and third rows are copied in twoRows matrix. Notice the indexing convension. It is a zero-based index, so the index of first, second and third rows are zero, one and two. However, in the rowRange function, it takes rows from the first argument (1) to one less than the second argument. Since we want the rows one and two to be copied; so we’ve given the values (1,3). We have copied this two rows in the first two rows MyMat by the copyTo method using the range argument (0,2). We need to remember one point here. copyTo function CAN NOT refer to itself. That is, you can not combine line 10 and 13 as follows.

MyMat.rowRange(1,3).clone().copyTo(MyMat.rowRange(0,2));
// Does Not Work!!

In Line 18, a block of the matrix is selected and set to 1.5. Output of this code is in line 15 of the program output. The program also shows the use of the diag method. It gives the diagonal entries of a matrix as a column vector. diag can take a integer as its argument. If its value is zero, it refers to the main diagonal. On the other hand, positive integers indicate the diagonals upper than the main one, and negatives indicate lower diagonals.

First thing that you need to do to work with opencv is to install it. If you are using Windows and Microsoft Visual Studio, I have a set of instructions to install it. However, if you are using Linux, Mac, Android or any other OS, you need to look for the instructions in OpenCV Wiki. Once you’ve successfully installed it, let’s jump into the coding part.

From OpenCV 2, it is possible to use the C++ coding style in OpenCV. It has different modules focused on different kinds of works. For example, “core” is mainly devote to the core processing like forming the matrix, doing some linear algebra etc. The “highgui” module deals with GUI related works such as creating windows etc. “ml” works with machine learning procedures. If you work with computer vision, you will eventually see that  it is heavily dependent on the machine learning field. Anyway, the header files for different components of OpenCV are located in <OpenCV>\include\opencv2\<ModuleName> folder. If you don’t want to bother to know which function belongs to which module, you can include only one header “opencv.hpp” which includes all the other header files. So, all the parts of OpenCV is added in your project.

Lets first see some basic input-output of a matrix. Write the following code in your MSVC project which you should configure according to the instructions specified here.

#include <opencv2/opencv.hpp>;
#include <iostream>;
int main(int argx, char** argv){
 float trainingData[3][3] = { {1, 2, 3}, {4, 5, 6}, {7, 8, 9}};
 cv::Mat MyFirstMat(3, 3, CV_32FC1, trainingData);
 std::cout<<MyFirstMat;
 std::cout<<"\nPress Enter to exit\n";
 std::cin.get();
}

When you’ll run this small program, you’ll see a matrix in the output screen. The entries in each row are separated by a comma. After every row, there is a semicolon. Actually, this is a way of writing matrices in many other languages including MATLAB. Look at lines 4 and 5 very carefully. In 4, you are defining a 3×3 array where you’re loading the entries of the matrix. Line number 5 is the main OpenCV command where you are creating an OpenCV matrix named “MyFirstMat”. Look at the constructor arguments carefully. The first two arguments indicate that the matrix contains 3 rows and 3 columns. The third argument refers to the type of the matrix. There is an OpenCV standard for specification of the types of OpenCV matrices. After the first common “CV” part, the number 32 means it is a 32 bit number (i.e. each entry in the matrix will be constituted with 32 bits). The “F” means it is a “float” type (i.e. contains decimal point). Some other common type specifiers are “64F”, “32S”, and “8U” representing “double”, “int”, and “uchar” respectively. In the last part, C1 means it has only one channel. In other words, you can write only one number in each entry. If it would be C3 then you could write 3 numbers. For example, in an RGB image, for each pixel you need to write 3 numbers for red, green and blue channel. In the last argument, we just fed the 2D array into the matrix. Notice that we specifically mentioned the scope using a scope resolution operator “:”. This is because, many functions in OpenCV matches with the functions of standard library “std”. So, it is a good practice to specify the scopes in every case.

Now let us do an experiment. Replace line number 4 by this line:

float trainingData[9] = { 1, 2, 3, 4, 5, 6, 7, 8, 9}

It is giving the same result, right? This is because, when the data is store into a matrix, it is stored as just by writing the bits one after another. It is the responsibility of the first three arguments (number of rows, number of columns and the datatype) to interpret the bit sequence as a human understandable form. Unlike MATLAB, OpenCV vectorizes a matrix row-wise first.

Now let us see some other examples of matrix input. The following code will also do the same as previous example:

Mat cameraMatrix = (Mat<double>(3,3) << 1000, 0, 320, 0,
 1000, 240 , 0, 0, 1);

Sometimes, it is necessary to dynamically input the data after initializing the headers. This can be done as below.

int main(int argx, char** argv){
 cv::Mat MyMat = cv::Mat::zeros(3,3,CV_64FC1);
 for(int i=0;i<3;i++)
  for (int j = 0;j<3;j++)
   MyMat.at<double>(i,j)=j+i*3+1;
 std::cout<<MyMat;
 std::cout<<"\nPress Enter to exit";
 std::cin.get();
}

It gives the same output as earlier example. But the difference is, this time you don’t need to specify all the values in coding time. The matrix entries gets assigned in run-time (i.e. dynamically) by evaluating a mathematical function. Notice, in this code, the matrix is initialized with zeros using a static function named “zeros” (all elements will be zero). You can also initialize it with the “ones” (all elements will be one), or the “eye” (identity matrix). No matter how the initialization is done, the two for loops are actually replacing each element with the value of j+i*3+1. The “at” in line 5 is a template for accessing  the elements of the matrix. Since it is a template, you need to specify the data type. This method can be used to both input and output of a matrix element. You need to be careful in deciding whether you are putting (i,j) or (j,i) as the arguments of the “at” function. Since OpenCV assigns the values of a row first, the “for” loop defining the column variable (j in this case) should immediately precede the line of at function. But don’t worry much, if you get the matrix in wrong orientation, you can always transpose it using the t() function. You can test it by inserting the following line in between lines # and # in the previous code .

 MyMat.t();

You can use the following code to load a matrix from a csv file.

 CvMLData Data;
 Data.read_csv("test.csv");
 csvContent = Data.get_values();

Any OpenCV structure can be saved in xml/yml file. If you are interested to do so, you can see here. I’ll not discuss it here.

Now, you know the basic methods to load a matrix and show it in screen. You can also do element by element operation since you can access matrix elements. However, there are several convenience function. For example, a matrix can be displayed in an output stream (or file-stream) using the << operator. In next post, we’ll see some basic operations in opencv. We’ll also see some linear algebra operations.

For example, you can read, show and write images using the imread, imshow and imwrite commands respectively.

#include <stdio.h>
#include <opencv2\opencv.hpp>
#include <string>
#include <fstream>

using namespace std;

int main(int argc, char* argv[])
{
	// ============= Read, Show, Write Image ===================
	// CAUTION!!! This statement requires a jpg file
        //in the current directory
	cv::Mat img = cv::imread("test.jpg");

	if(img.data!=NULL){
	   cv::Mat scaledimg(img.rows/img.cols*640,640,img.type());
		cv::resize(img,scaledimg,scaledimg.size());

		cv::namedWindow("Image");

		cv::imshow("Image",scaledimg);
// See what happens if you don't use this
		cv::waitKey();
		cv::imwrite("test_scaled.jpg",scaledimg);
	}
	return 0;
}

When the question of video comes, OpenCV has a one stop solution named VideoCapture. It can capture video frames from both a video or a webcam.

#include <stdio.h>
#include <opencv2\opencv.hpp>
#include <string>
#include <fstream>

using namespace std;

int main(int argc, char* argv[])
{
	// ============== Capture Camera Frame ====================
	cv::Mat frame;
	// Capturing data from camera
        // Caution: It requires webcam to be attached
	cv::VideoCapture cap(0);
        // you can also use
        // cv::VideoCapture cap("video filename");
        // to capture the frame from a video instead of webcam
	if(!cap.isOpened())
	  printf("No Camera Detected");
	else{
		cv::namedWindow("Webcam Video");
		for(;;){
			cap >> frame; // get a new frame from camera
			cv::imshow("Webcam Video",frame);
			if(cv::waitKey(30) >= 0) break;
		}
	}
	return 0;
}

Note: This post assumes you have OpenCV 2.3 installed. If not, please check the following post
http://www.itanveer.com/2011/installing-opencv-231/

// OpenCV_Test.cpp : Defines the entry point for the console application.
//

#include <stdio.h>
#include <opencv2\opencv.hpp>
#include <string>
#include <fstream>

using namespace std;

// I wrote this method to display matrix contents. It is only for
// double (64 bit) data type. I shall write a more general version
// in near future
void dispMat(CvMat *N, string varName){
    double *data = N->data.db;
    int rowNum = N->rows;
    int colNum = N->cols;

    printf("\n========= Matrix Display System ==============\n");
    printf("(%d,%d) %s \n",rowNum,colNum,varName.c_str());
    for(int i=0;i<rowNum;i++){
        if(colNum>10)
            printf("Row No=%d\n",i+1);
        for(int j=0;j<colNum;j++)
            printf("%0.2f\t",data[i*colNum+j]);
        printf("\n");
    }
    printf ("=================================================");
    printf("\n");
}

int main(int argc, char* argv[])
{
    int a=0;

    double c[] = {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16};
    double d[] = {2,1,6,-4,0.5,4,9,1,9,-1,-0.3,0.7,3,2,1,1.5};

    // Declaring Matrix as a Pointer and initialization
// This is the old style of OpenCV Matrices
    CvMat* M = cvCreateMat(4,4,CV_64FC1);
    cvInitMatHeader(M,4,4,CV_64FC1,d);
    dispMat(M,"M = ");

    // Declaring Matrix as variable and initialization
    CvMat N = cvMat(4,4,CV_64FC1,c);
    dispMat(&N,"N = ");

    // Declaring Matrix as pointer without initialization
    CvMat* M2;

    //Cloning a Matrix
    M2 = cvCloneMat(M);
    dispMat(M2,"M2 = ");

    //indirect access to matrix element
    //get
    printf("\n(2,3)th element in M2 = %0.2f\n",cvmGet(M2,2,3));
    //set
    cvmSet(M2,2,3,90);
    printf("(2,3)th element is now set to = %0.2f\n",
	cvmGet(M2,2,3));

    cout<<"\nPress Enter to continue\n";cin.get();
    // ================================================

    //Set zero
    cvSetZero(M2);
    dispMat(M2,"\nM2 (cvSetZero)\n");

    //Set Identity
    cvSetIdentity(M2);
    dispMat(M2,"\nM2 (cvSetIdentity)\n");

	cout<<"\nPress Enter to continue\n";cin.get();

    // ============ OpenCV 2 Test =====================
    cv::Mat A = cv::Mat(M,true);
    cv::Mat J = cv::Mat(4,1,CV_64FC1,cv::Scalar(1));

    cv::Mat B = (J.t())*(A);

    CvMat AA = A;
    dispMat(&AA,"OpenCV 2 Variable: A");

    AA = J;
    dispMat(&AA,"OpenCV 2 Variable: J");

	// Try this: B  = (J.t()*A).t();
    B  = (J.t()*A);
	B = B.t();
	AA = B;		// Also try this: AA = B.t();
    dispMat(&AA,"OpenCV 2 Variable:(J'*A)'");

    cout<<"\nPress Enter to continue\n";cin.get();
    // =========== Predefined openCV 2 Matrices ========
    A = cv::Mat::zeros(5,2,CV_64FC1);
    AA = A;
    dispMat(&AA,"Predefined matrices: zeros(5,2,cv_64fc1)");

    A = cv::Mat::ones(5,2,CV_64FC1);
    AA = A;
    dispMat(&AA,"Predefined matrices: ones(5,2,cv_64fc1)");

    A = cv::Mat::eye(5,5,CV_64FC1);
    AA = A;
    dispMat(&AA,"Predefined matrices: eye(5,5,cv_64fc1)");

    cout<<"\nPress Enter to continue\n";cin.get();
    // =========== Predefined openCV 2 Matrices ========
    printf("\n\n Element (1,2) = %f",A.at<double>(2,1));

    // === Writing opencv Matrix (M2) to and from Filestream =====

    ofstream ofs("d:\\test.dat",ios::out|ios::binary);

    dispMat(M2,"Matrix before writing to file");

    double p =5;int s=3;
    double* m2Dat = M2->data.db;

    ofs<<p<<endl<<s;
    ofs.write((char *) m2Dat,M2->cols*M2->rows*sizeof(double));

    ofs.close();

    p=0;s=0;
    m2Dat[0]=-1;
    m2Dat[1]=-1;
    m2Dat[2]=-1;
    m2Dat[3]=-1;
    m2Dat[4]=-1;
    m2Dat[5]=-1;
    m2Dat[6]=-1;
    m2Dat[7]=-1;
    m2Dat[8]=-1;
    m2Dat[9]=-1;
    m2Dat[10]=-1;
    m2Dat[11]=-1;
    m2Dat[12]=-1;
    m2Dat[13]=-1;
    m2Dat[14]=-1;
    m2Dat[15]=-1;

    dispMat(M2,"Matrix after changing data");

    ifstream ifs("d:\\test.dat",ios::in|ios::binary);

    ifs>>p>>s;
    printf("%0.3f\n",p);
    printf("%d\n",s);
    ifs.read((char *)m2Dat,M2->cols*M2->rows*sizeof(double));

    dispMat(M2,"Matrix after Loading again");

    CvMat* M3 = cvCreateMat(4,4,CV_64FC1);
    cvInitMatHeader(M3,4,4,CV_64FC1,m2Dat);
    dispMat(M3,"Matrix reading from file");

    ifs.close();

    // Releasing Matrix Handle
    cvReleaseData(M);
    cvReleaseData(M2);
    cvReleaseData(M3);

    return 0;
}

Note: This post assumes you have OpenCV 2.3 installed. If not, please check the following post
http://www.itanveer.com/2011/installing-opencv-231/

Preparation Step: Install Visual Studio 10.

Stage One: Installing OpenCV

1. Download OpenCV 2.3.1 Superpack from the following link: http://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.3.1/.
2. Run the exe file and extract the contents in C:\OpenCV231. If the contents (i.e. the folders named 3rdparty, android etc) are copied in a folder named “opencv” inside C:\OpenCV231 then copy all of them to C:\OpenCV231. (I’ll describe all the later steps assuming that the files has been extracted here. If not, please modify the instruction accordingly.)
   

   Figure 1: Selecting the Generator

   

   Figure 2: cmake window

3. Download and install cmake from http://www.cmake.org/
4. Run the Graphical User Interface of Cmake (cmake-gui).
5. In the field named “Where is the source code:” white c:\opencv231. In the field named “Where to build the binaries” specify C:\OpenCV231\msvc. Now click “configure”.
6. A dialog box will appear (Figure 1) asking you to specify the generator for the project. Select Visual Studio 10 from the dropdown list. Select “Use default native compiler”. Now click “Finish”. It may ask for your permission to create a new folder named msvc. Just allow to do it. It
7. Now the cmake window will look something like Figure 2. In this dialog, check the box named “install_c_examples”. Click the “Generate” button. This will make Visual Studio solutions in the C:\OpenCV231\msvc folder.
8. Open OpenCV.sln. Select “Debug” in the solution configuration listbox. Now press F7 to build the solution. If all the projects are successfully built (3 projects may be skipped, don’t worry about that) it will show such message. Now do the same by selecting “release” in the solution configuration listbox.
9. Copy the “lib” and “bin” folders from C:\OpenCV231\msvc to C:\OpenCV231. Also copy the contents of “include” folder from C:\OpenCV231\build to C:\OpenCV231\include
10. Now we need to add the following 2 folders in environment variables:
    C:\OpenCV231\bin\Release
    C:\OpenCV231\bin\Debug
    To do this, go to Control Panel > System > Advanced System Settings > Environment Variables and append these two folders into the “path” variable. Remember to separate the entries by semicolon. Click ok.

Compatibility Issue:

I have found the following backward compatibility issue while moving from opencv 2.1 to 2.3.1

If M be a cv::Mat then if you have a command M.inv(CV_SVD_SYM) then you have to change it to M.inv(cv::DECOMP_SVD)

Update: OpenCV 2.4.0 is released. It has very similar installation process except the lib filenames. Release filenames:

opencv_calib3d240.lib
opencv_contrib240.lib
opencv_core240.lib
opencv_features2d240.lib
opencv_flann240.lib
opencv_gpu240.lib
opencv_haartraining_engine.lib
opencv_highgui240.lib
opencv_imgproc240.lib
opencv_legacy240.lib
opencv_ml240.lib
opencv_nonfree240.lib
opencv_objdetect240.lib
opencv_photo240.lib
opencv_stitching240.lib
opencv_ts240.lib
opencv_video240.lib
opencv_videostab240.lib

And Debug Filenames

opencv_calib3d240d.lib
opencv_contrib240d.lib
opencv_core240d.lib
opencv_features2d240d.lib
opencv_flann240d.lib
opencv_gpu240d.lib
opencv_haartraining_engined.lib
opencv_highgui240d.lib
opencv_imgproc240d.lib
opencv_legacy240d.lib
opencv_ml240d.lib
opencv_nonfree240d.lib
opencv_objdetect240d.lib
opencv_photo240d.lib
opencv_stitching240d.lib
opencv_ts240d.lib
opencv_video240d.lib
opencv_videostab240d.lib

The Use of Active Appearance Models

With Professor Jeffrey Cohn

Renowned researcher of psychology Professor Dr. Jeffrey Cohn, in his presentation “Machine Learning for Affective Computing: Findings and Issue” showed how Active Appearance Model has paved the way for some of the experiments which were not possible to conduct earlier. For example, the capability of Active Appearance Models to accurately model and tack a person’s head movement and facial expressions is utilized to generate almost real time (~0.5 sec delay) avatars. These avatars are used in interaction between two people in several psychological experiments. For example, it was shown that, people can interact as easily with an avatar as a normal human being. However, these interactions can be manipulated by reducing the amount of expressions (which can be done by computationally reducing the expressions expressed by the avatars) or by immobilizing the head movement or by changing the gender of the avatar (A male showing a female avatar and vice versa) etc. In those cases, people do not identify such interactions as spontaneous human-human interactions. Prof Cohn also showed the effect of smile dynamics of faces i.e. how the “spontaneity” in smile is related with the temporal evolution. In a later presentation named “FAST-FACS: A Computer-Assisted System to Increase Speed and Reliability of Manual FACS Coding” Jeff Cohn also showed how the manual landmark annotation effort can by reduced by 40% through the use of some semi-automatic annotation process.

Human centric Machine Learning

Some exciting achievements are possible to make by incorporating humans in different aspects of machine learning. This point is underscored by the invited speaker from Microsoft research, Ashish Kapoor, in his presentation “Human-Centric Machine Learning”. He mentioned that, it is possible to use human interactions more than just data labeling. He showed several of his works on how it is possible to use human interactions to rectify the decision boundaries of classifiers, incorporate experience and use in many other exciting ways.

More on Smile Dynamics

More research on the types of smile has been shown in the paper “Are You Friendly or just Polite? – Analysis of smiles in spontaneous face-to-face interactions” presented by Mohammed Hoque, Louis-Philippe Morency and Rosaline Picard. It has been shown that most of the spontaneity information lies on what time it takes for a smile to rise to the peak from the onset and decay to the offset from the peak. Mohammed Hoque mentioned that MIT media lab will release the dataset that has been used in this work. I believe it will give us a rare opportunity to access some spontaneous dataset just for free.

Interactions and Socialization

With Professor Rosaline Picard, the founder of Affective Computing

This conference gave me a wonderful opportunity to see and discuss with the leading researchers in affective computing. I talked personally with Jeffry Cohn, Rosaline Picard, Peter Robinson, Louis-Philippe Morency and asked few questions regarding current trends of research in affective computing. In a conversation with Prof Cohn on the channels of emotion, he mentioned that he is a bit skeptical on how far it is possible to infer emotion appropriately from physiological channels. He mentioned two reasons for that. Firstly, physiological responses are relatively slower than the response showed in facial expressions. Secondly, sensors required to measure physiological signals can alter the emotions to some extent.

Professor Rosaline Picard highly appreciated the project concept of blind emotion. She mentioned that it is possible to help blind people in great extent to perceive others emotion. MIT Media lab did some preliminary investigations on these issues. She specifically pointed that it will not be a good idea to convey the facial expressions (like smile, frown etc.) to the people who are congenitally blind. Rather, it is better to convey some inferred information which makes more sense to them. For example, whether other people are interested on what a blind person is speaking etc. Prof Picard also pointed on a possible difficulty that their lab faced. She mentioned that it becomes very difficult to track and analyze a person’s face with the inherent head movement of the wearer. She also welcomed any help and collaboration that may be asked from our side.

Dr. Louis-Philippe Morency showed an extra ordinary enthusiasm to our effort on analyzing

With Professor Peter Robinson

facial expressions and works with active appearance model. He mentioned that he is going to release a more robust version of Constrained Local Model (CLM) based face tracker. This tracker will be released in public domain. He also expressed his ideology on how research related data and tool should be disseminated for public use. He also gave word to entertain our request for this face tracker as much as possible.

It is apparent from a long growing mass of evidences that the part of a brain responsible for processing information from a particular sensory channel gets recruited for other activities after loss of that particular channel. This phenomenon is referred to as “neuroplastic” behavior of brain. When it contributes to the improvement of other sensory channels, it is called “crossmodal neuroplasticity”. This paper refers to a considerable amount of reported evidences on crossmodal neuroplasticity following sensory deprivation and came to some conclusions based on those observations.

It particularly discussed about the following aspects of neuroplasticity. Firstly, it refers to a significant amount of reported experiments and their results on the effect of loss of vision and auditory capabilities. Analyzing these experiments, a conclusion was made that the loss of a sensory channel translates into an improved or at least an equal performance in behavioral and sensory tasks. Experimental proofs have been referred in this paper where it is reported that blind individuals showed equal or superior performance in tactile-discrimination thresholds, auditory-pitch discrimination, spatial sound localization, speech discrimination, verbal recall etc. In addition, it has been discussed that electroencephalographic and neuro-imaging technologies provide strong evidence on an altered association of brain areas for proficient utilization of other sensory channels. Similar works have also been showed for deafness and loss of multiple sensory channels.

Other than collecting evidences of crossmodal neuroplasticity, this paper also discussed on reported experiments which can explain the potential causes and mechanisms under this phenomenon. A potentially viable cause of recruitment of cortical areas followed by a sensory loss is thought to be the direct connection of the neurons to the intact sensory area. This paper also discussed on the endeavors made for determining the critical age for crossmodal plasticity. The recent experiments showed that neuroplasticity can occur in a much later age than that were presumed earlier.

Lastly, the paper discussed that in spite of the compensatory behavioral gain, the neuroplasticity cannot be viewed as universally adaptive. It may also have unintended consequences. Following the rehabilitative training, the neuroplastic changes can undermine the ability of the cortex to return back to its primary function.

Amir Amedi 1,2, William Stern 1, Joan A. Camprodon 1, Felix Bermpohl 1,3, Lotfi Merabet 1, Stephen Rotman 1, Christopher Hemond 1, Peter Meijer 4 and Alvaro Pascual-Leone

This paper reports some functional characteristics of a part of brain known as the lateral-occipital tactile-visual (LOtv) area. Traditionally, this part is known to be activated when a person observes the shape of an object through vision or touch. In this paper, it is claimed that this part of brain is responsible for analyzing shape of an object regardless of the modality of the incoming signal. This assertion has been made based on an experiment where the authors analyzed the functional characteristics of the LOtv area in response to sensory substitution soundscape.

VOICE is a computer program that converts the raw pixel intensities of a picture into a soundscape. There are some blind and sighted expert users who can perceive the shapes of objects by listening to the soundscapes. Neural images of these people’s brain reveal that when they try to perceive the shape of an object from the soundscape, their LOtv area activates. Moreover, this happened only to the people who were trained to interpret the soundscape and can extract shape information from it. People, who associate a particular soundscape to a particular shape of an object, do not display any activity in this area.

The authors have proposed a hypothesis based on this observation. They claimed that, the LOtv processes the shape information regardless the modality of sensory signal.

After the Iraq war, veterans who lost there limbs, sometimes felt unnatural sensations in their lost limbs. How would you explain such phenomenon? For a long time, it was considered as a hoax. Neuroscientists could not accept this as a true sensation because for every sensation there should have some neurons carrying small electric signals to the brain. In the case of phantom limb, this is not possible. Then what?

Dr. Ramachandran made a pioneering work in explaining phantom limb syndrome while providing treatment to one of his patients named Derick Sting. Derick used to feel vivid presence of his left arm in a strange way. When he used to go for a shave, he felt sensation in the left arm which actually does not exist. It was found in some simple experiment that when Derick’s left cheek is touched, he feels that sensation both in his cheek and in the left arm. Dr. Ramachandran got very interested. He knew that different parts of our body are mapped to some part of our brain. For example, the entire left part of the body is mapped to a strip on the right part of the brain known as somatosensory cortex. So he hypothesized that, when a limb gets amputated it looses its connection from brain and that part of the brain does not receive any signal anymore. Now, without any signal, this part becomes thirsty for some sensory signal and gets more and more sensitive to any kind of electric impulses. When some electronic impulses come to the nearby region, it actually catches that signal and gives a false signal to brain regarding a sensation on the person’s lost limb. Therefore, the postulation was the brain undergoes a massive rewiring after the amputation. Dr. Ramachandran got some scornful comments regarding his hypothesis. One strong argument was based on the dogma that the brains wiring is permanent and does not change once it is developed in the fetus. However, Ramachandran’s hypothesis got support when neural images provided support on his side.

Amputation can alter brain’s work not only in receiving signal but also transmitting. A patient used to feel extreme pain like clinching his right hand too much. But there is no right hand in his body. Ramachandran explained this phenomenon like this: When we clinch our hand, it actually sends some feedback to the brain which damps brains signal to clinch. But if the limb is amputated, it does not give any feedback to the brain and as a result brain sends more and more signal which eventually leads to extreme pain.

This problem was solved by the use of a technique named as “mirror-box“. A mirror box is a kind of apparatus where you see the image of your one hand in place of your another hand. Using this box, if a person moves his left hand, he gets an illusion that his right hand is moving. The patient was advised to put both his hands into the mirror-box and move both hands. Through this, he got a visual clue that his hand his moving. This visual information also acted as a feedback to the brain and the pain started to subside. This proved Ramachandrans theory and a very intriguing assertion that even a pain can be a construct of mind.

Interesting Follow Ups

1. Ramachandran, V. S., Brang, David and McGeoch, Paul D.(2009) “Size reduction using Mirror Visual Feedback (MVF) reduces phantom pain“, Neurocase, 99999:1 DOI 10.1080/13554790903081767
2. Ramachandran, VS & Hirstein, W (1998). The perception of phantom limbs: The D. O. Hebb lecture. Brain, 121, 1603-1630.