How to add UDF permanently to Hive

Tags

, , ,

Hive

Hive

To learn more about Hive UDF (https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF)

I recently developed a bunch of hive UDF’s and to call the function I have to add the jar files and create the temporary functions for every hive session. I started digging in to the code to find out If I can modify the java files and then rebuild hive. This post describes how I did it.

1. Download the source code from

 http://www.apache.org/dyn/closer.cgi/hive/

0.10.0 is the most stable version of hive when I’m writing this blog post.

2. Extract the tar ball and copy your udf  java files to

 \hive-0.10.0\src\ql\src\java\org\apache\hadoop\hive\ql\udf

if the udf is a generic udf copy to it

 hive-0.10.0\src\ql\src\java\org\apache\hadoop\hive\ql\udf\generic

3. Before you copy the java files to the ql folder you have to change the package of the java files to

 package org.apache.hadoop.hive.ql.udf.generic;  or package org.apache.hadoop.hive.ql.udf;

4. Now that you copied files to the udf folder you have to tell hive on how to find these functions. To do this you have change FunctionRegistry.java. You can find FunctionRegistry.java in

 \hive-0.10.0\src\ql\src\java\org\apache\hadoop\hive\ql\exec

5. You have to make the following changes to FunctionRegistry.java

i)  import the udf class

 ex: import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFToMap;

ii)  if its a UDAF

 registerGenericUDAF("to_map", new GenericUDAFToMap());

else

 registerGenericUDF("rank", GenericUDFRank.class);

6. Once you have made all the changes navigate to the src folder and build using ant ( ant package) once its build you will have a build folder. Even if the build fails (To completely build hive you need thrift compiler and many others installed, as long as it build hive-exec-*.jar you are good). It is recommended to deploy the entire build but only the hive-exec-*.jar  needs to be replaced.

I also observed that when I build hive on the windows machine and copy the jar to centos box it fails.

Hope this helps some one. Let me know if you have any trouble. You can reach me at @abhishek376 on twitter

Upload a picture using Andriod and PHP

Tags

, , , ,

andriod

I was working on Android app this weekend where I take pictures with the phone camera and upload them to a different server. I had a hard time figuring out saving the image to SD Card and accessing the picture back to upload it to the server. I thought this would be useful for somebody else like me trying to play with android camera.

1) Firstly we have to create a directory where we want to save the images in the sd card.

File dir = new File("/sdcard/foldername/");
 dir.mkdirs();

2) Next we set the file name and path

File file = new File(Environment.getExternalStorageDirectory().getPath() + "/foldername/" + imgName);
Uri imageUri = Uri.fromFile(file);

3) Create a new Intent with andriod.provider.MediaStore.ACTION_IMAGE_CAPTURE

Intent intent = new Intent(android.provider.MediaStore.ACTION_IMAGE_CAPTURE);

4) This is the most important step don’t forget to add the file uri in the intent put extra and then Start Activity for Result

intent.putExtra(MediaStore.EXTRA_OUTPUT, imageUri);
startActivityForResult(intent, 0);

Once you take the picture and click ok you will be redirected back to the activity that started the camera activity

protected void onActivityResult(int requestCode, int resultCode, Intent data) {
 String pathToOurFile = Environment.getExternalStorageDirectory().getPath() + "/foldername/" + imgName;
    //Upload the image to the server
}

To upload the image, in my case I’ve used a PHP script in the server to handle my upload. Don’t forget to run the upload file in the new thread.

protected void uploadFile()
 {
 HttpURLConnection connection = null;
 DataOutputStream outputStream = null;
 //DataInputStream inputStream = null;
String pathToOurFile = Environment.getExternalStorageDirectory().getPath() + "/foldername/" + imgName;
 String urlServer = "http://path to php file";
 String lineEnd = "\r\n";
 String twoHyphens = "--";
 String boundary = "*****";
int bytesRead, bytesAvailable, bufferSize;
 byte[] buffer;
 int maxBufferSize = 1*1024*1024;
try
 {
 FileInputStream fileInputStream = new FileInputStream(new File(pathToOurFile));
URL url = new URL(urlServer);
 connection = (HttpURLConnection) url.openConnection();
// Allow Inputs & Outputs
 connection.setDoInput(true);
 connection.setDoOutput(true);
 connection.setUseCaches(false);
// Enable POST method
 connection.setRequestMethod("POST");
connection.setRequestProperty("Connection", "Keep-Alive");
 connection.setRequestProperty("Content-Type", "multipart/form-data;boundary="+boundary);
outputStream = new DataOutputStream( connection.getOutputStream() );
 outputStream.writeBytes(twoHyphens + boundary + lineEnd);
 outputStream.writeBytes("Content-Disposition: form-data; name=\"uploadedfile\";filename=\"" + pathToOurFile +"\"" + lineEnd);
 outputStream.writeBytes(lineEnd);
bytesAvailable = fileInputStream.available();
 bufferSize = Math.min(bytesAvailable, maxBufferSize);
 buffer = new byte[bufferSize];
// Read file
 bytesRead = fileInputStream.read(buffer, 0, bufferSize);
while (bytesRead > 0)
 {
 outputStream.write(buffer, 0, bufferSize);
 bytesAvailable = fileInputStream.available();
 bufferSize = Math.min(bytesAvailable, maxBufferSize);
 bytesRead = fileInputStream.read(buffer, 0, bufferSize);
 }
outputStream.writeBytes(lineEnd);
 outputStream.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);
// Responses from the server (code and message)
 @SuppressWarnings("unused")
 int serverResponseCode = connection.getResponseCode();
 @SuppressWarnings("unused")
 String serverResponseMessage = connection.getResponseMessage();
fileInputStream.close();
 outputStream.flush();
 outputStream.close();
 }
 catch (Exception ex)
 {
 //Exception handling
 }
 }

Leave a comment if you need some help.

Running OpenCV 2.2 using PHP on Amazon Cloud Free Tier

Tags

, , , , , , ,

opencv

amazone2

What is OpenCV ?

OpenCV means Intel® Open Source Computer Vision Library. It is a collection of C functions and a few C++ classes that implement many popular Image Processing and Computer Vision algorithms.

Welcome – OpenCV Wiki

What are Amazon web services ?

Amazon Web Services (AWS) is a collection of remote computing services (also called web services) that together make up a cloud computing platform, offered over the Internet by Amazon.com. The most central and well-known of these services are Amazon EC2 andAmazon S3.

Why this tutorial ?

If your are like me who had a lot of trouble installing opencv on your windows or linux machine or having a lot of trouble getting opencv work on web. In this tutorial I will cover how to install opencv 2.2 on Amazon cloud using free tier usage and call opencv functions using PHP5 . Amazon gives any new user 750 hours of Linux micro instance per month for 12 months. Which is 1 year of free linux instance on the amazon cloud.

For the free tier usage check this link : http://aws.amazon.com/free/faqs/

Setting Up Amazon EC2

1. Sign up on http://aws.amazon.com/free/ (You need  a credit card to signup)

2.  After you sign up you can log in to your aws console https://console.aws.amazon.com/ec2/home

3.  Launch a instance

4. You need to now choose the operating system on your instance we will choose Ubuntu 10.04

5.  In the instance details make sure you choose Micro instance which is eligible for amazon free tier

6. You need to select key-pair. Create a new key pair and download the key. You will need the key to SSH in to the instance.

7. Choose the security group and make you have a rule to allow HTTP (Port 80)

8. And finally launch instance.

How to connect to your linux instance :

9. Right click your instance to and choose connect to get the public DNS of the instance.

10. In this tutorial I’m using winscp and putty to connect to SSH.

You need to get putty gen from link below :

http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

Go to File-> Choose load private key -> Select *.pem file (key of your instance which you downloaded while creating a new instance)  and save the private key.

11. Open winscp . Create a new login

Choose host as your public DNS.

Username :  ubuntu

choose you private key file. You don’t need any password.

That should get you in.

Installing OpenCV 2.2

1. First you need to install all the dependencies

sudo apt-get install build-essential libgtk2.0-dev libjpeg62-dev libtiff4-dev libjasper-dev libopenexr-dev cmake python-dev python-numpy libtbb-dev libeigen2-dev yasm libfaac-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev

2. To get the video working properly we need to compile ffmpeg. If you are not worried about the videos you can ignore this step if you get some errors

cd ~
wget http://ffmpeg.org/releases/ffmpeg-0.7-rc1.tar.gz
tar -xvzf ffmpeg-0.7-rc1.tar.gz
cd ffmpeg-0.7-rc1
./configure --enable-gpl --enable-version3 --enable-nonfree --enable-postproc --enable-libfaac --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvorbis --enable-libxvid --enable-x11grab --enable-swscale --enable-shared
make
sudo make install

3. Install OpenCV. This will take a while because of low CPU and RAM (613MB)

cd ~
wget http://downloads.sourceforge.net/project/opencvlibrary/opencv-unix/2.2/OpenCV-2.2.0.tar.bz2
tar -xvf OpenCV-2.2.0.tar.bz2
cd OpenCV-2.2.0/

4. Next we have to generate Makefile using CMAKE. We need to configure cmake

cmake -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=OFF -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON .
make
sudo make install

5. Next we have to configure OpenCV First, open the opencv.conf file with the following code:

sudo gedit /etc/ld.so.conf.d/opencv.conf
and copy
/usr/local/lib

6. To configure the library

sudo ldconfig
sudo gedit /etc/bash.bashrc
Add these two lines at the end of the file
PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig
export PKG_CONFIG_PATH

7. To Run the demos. Replace x with with appropriate file permission (ex chmod -R 755 build_all.sh)

cd ~/OpenCV-2.2.0/samples/c
chmod +x build_all.sh
./build_all.sh

Once compiled you can run a demo by calling ./demofilename

Installing Apache2 and PHP on ubuntu

sudo apt-get install apache2
sudo apt-get install php5
sudo apt-get install libapache2-mod-php5
sudo /etc/init.d/apache2 restart

Once apache is installed you check by opening http://localhost to see if apache is running. To manually start apache

sudo etc/init.d/apache2 start

All the file are located in /var/www

You need to make the www writable

sudo chown -R username directorypath

create a new file in the www folder called opencv.php

Running OpenCV using PHP

Now that you have apache and PHP installed. You can use exec function of the PHP to call the opencv compile executables.

You can find the samples in opencv/samples/c . You need to build them using ./build_all.sh

You can copy the compiled demo file to www dir. For example you have compile testcv.cpp to testcv

In the opencv.php file

 <?php
exec('./testcv',$results);
foreach($results as $l)
{
  echo $l;
}
?>

To run the file

To publicly access the file you can use http://{your amazon public dns}/opencv.php

Leave a comments if you have any trouble.