Apache Spark | Run first spark program

-
In this post we are going to run sample spark program. If you want to setup spark locally go to this post. In the last post we created slave node with 2 cores and 2g memory. This worker node will be used by master to run your spark jobs.

Spark installations provides some sample programs to run on cluster. This can be found at below location.
rohan@rohan:/opt/spark/spark-2.3.1-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples$ ll
total 64
drwxrwxr-x 6 rohan rohan 4096 Jun 1 22:49 ./
drwxrwxr-x 3 rohan rohan 4096 Jun 1 22:49 ../
-rw-rw-r-- 1 rohan rohan 4506 Jun 1 22:49 JavaHdfsLR.java
-rw-rw-r-- 1 rohan rohan 4683 Jun 1 22:49 JavaLogQuery.java
-rw-rw-r-- 1 rohan rohan 4303 Jun 1 22:49 JavaPageRank.java
-rw-rw-r-- 1 rohan rohan 1943 Jun 1 22:49 JavaSparkPi.java
-rw-rw-r-- 1 rohan rohan 2721 Jun 1 22:49 JavaStatusTrackerDemo.java
-rw-rw-r-- 1 rohan rohan 3477 Jun 1 22:49 JavaTC.java
-rw-rw-r-- 1 rohan rohan 1968 Jun 1 22:49 JavaWordCount.java
drwxrwxr-x 2 rohan rohan 4096 Jun 1 22:49 ml/
drwxrwxr-x 2 rohan rohan 4096 Jun 1 22:49 mllib/
drwxrwxr-x 4 rohan rohan 4096 Jun 1 22:49 sql/
drwxrwxr-x 2 rohan rohan 4096 Jun 1 22:49 streaming/
view raw SparkExample.sh hosted with ❤ by GitHub
We will be running JavaWordCount program locally in spark cluster.


Spark programs can be pushed to cluster using  ./bin/spark-submit script. More info on official website.

Place sample word.file under any dir(/opt/spark/spark-poc/words.txt for this example). Now run below command.
rohan@rohan:/opt/spark/spark-2.3.1-bin-hadoop2.7/bin$./spark-submit --class "org.apache.spark.examples.JavaWordCount" --master spark://rohan-Latitude-E5470:7077 /opt/spark/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar /opt/spark/spark-poc/words.txt
view raw SparkExecuteCmd.sh hosted with ❤ by GitHub

As you can see, spark-submit takes below arguments.

--class [main-class of spark program]
--master [master node url]
[app-jar]
[app command-line arguments]
Check Spark master UI to see status of spark program.

Comments

Post a Comment

Popular posts from this blog

Spring | Using TIBCO EMS with Spring framework

-
TIBCO EMS can be use with Spring Framework to support message publishing and subscription. Spring provides JMS template to send JMS messages and @JmsListener/ MessageListener to subscribe it. Steps 1 : Create new project in eclipse or IntelliJ Idea Create new project with Spring dependency. We can use maven to add all required libs for this project. Maven takes care of adding referred java libs to build path. TIBCO JMS jars need to be added to build path. Pom.xml will be as follows. As you can see in above pom file we have added TIBCO JMS and Spring JMS dependency. Step 2 : Create JMS bean configuration file(Jms-conf.xml) Next we will create bean configuration file. It will set JNDI and JMS properties. We can use same JMS listener class to subscribe multiple queues and topics. Spring bean configuration take care of JMS transport setup. Env variables values can be provided via properties file. Step 3 : Create JMS Queue/Topic sender and Listner ...
Read more

TIBCO | For Loop - Accumulate output

-
Image
In this post we will check how TIBCO for loop works. TIBCO provides Group Activity to iterate over processes or activities. Loops can be different types eg. For Loop, While util loop, if loop, repeat on error etc. TIBCO group can loop over sub-process and accumulate output in end activity to be used by next activities. TIBCO For loop with Accumulate output repeats over process/activity and return output of each iteration in output activity. This is helpful where we want to call sub-process repeatably with different input parameters and store each output in some variable.
Read more

TIBCO | JNDI Server & JMS Instance creation

-
Image
In this post we will be creating TIBCO JNDI server and other instance pointed to JNDI server. First lets see what is mean by JNDI. It is   Java API for a directory service that allows Java software clients to discover and look up data and objects via a name. TIBCO EMS uses JNDI concept to lookup for EMS objects and its relations in messaging service. While developing enterprise level TIBCO solution it is advise or say standard practice to create one JNDI server(also one backup JNDI server) and other instance servers according to the functionality or domains. In this tutorial we will be creating one JNDI server and two domain servers.
Read more