![\"\"](\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2017\/03\/spark-overview.png\")
<\/a><\/p>\nFollowing, we will consider the most important core concepts as well as the components SparkSQL, Spark Streaming and MLlib. We developed our Applications in Java, so the following examples are written in Java.<\/p>\n
Apache Spark Core<\/h3>\nResilient Distributed Datasets (RDD)<\/h4>\n
The core concept of Apache Spark is called Resilient Distributed Datasets (RDD). A RDD is a partitioned collection of elements which can be processed in parallel in a fault-tolerant manner. Fault-tolerant means, if any partition of an RDD is lost, it will be automatically executed again. You can create a RDD by transform an existing collection. Furthermore you can create RDDs by reading datasets from external data storages like distributed file systems or databases. We will show the basics of RDDs in a short example at the end of this section. [2]<\/p>\n
First RDD Example<\/h5>\n
We used in our Project Spark in Version 1.6.1, although the newest release is 2.1.0. The reason for that are on the one hand dependency issues to other Hadoop projects, on the other hand was this the installed version in our Hadoop cluster. We will cover this issue in more detail in the Big Data Engineering part. In the code below there are implemented some of the concepts described above.<\/p>\n
\/\/ create jspark configuration\nSparkConf conf = new SparkConf().setAppName(\"First Spark Test\");\n\/\/ create java spark context\nJavaSparkContext jsc = new JavaSparkContext(conf);\n\n\n \/\/ create test data\nList<String> data = Arrays.asList(\"This\",\"is\",\"a\",\"first\",\"spark\",\"test\");\n\/\/ create a simple string rdd\nJavaRDD<String> firstRDD = jsc.parallelize(data);\n\n\/\/ save to hdfs\nfirstRDD.saveAsTextFile(\"hdfs:\/\/..user\/syseng\/simple_spark_test\/example.txt\");\n\n\/\/ read text file\nJavaRDD<String> tfRDD = jsc.textFile(\"hdfs:\/\/..\/user\/syseng\/simple_spark_test\/example.txt\");<\/pre>\nFirst we have to create a Spark Configuration where we set the Name of the Application (line 2). Additionally, we need a Spark Context which expects the Configuration as Parameter (line 4). In line 7 we initialize a usual Java list. This list we pass as parameter to the parallelize-method in line 9 of the Spark Context object. This method transforms the list into an RDD, which means that the elements of the list will be partitioned. It is possible to pass a second parameter which indicates in how many partitions the List should be divided. If this program will be executed on a computer cluster, each host would process a partition for example. With the saveAsTextFile-method in line 11 the results will be saved in a file system for example a Hadoop File System. In the last line there is shown another way how an RDD can be created. The method textFile of the Spark Context reads a text file e.g. from a hadoop distributed file system and transforms the lines of the text file into an RDD of strings.<\/p>\n
RDD Operations<\/h4>\n
There are two types of operations you can apply on RDDs: transformations and actions. An transformation is applies a function to each element of an RDD and returns a new (transformed) RDD. An action actually calculates a result for the driver program. In Spark you build pipelines with higher order functions in the manner of functional programming like shown in the figure below. Each pipeline will be executed only if an action operation is called on a pipeline. That means if a Pipeline consists only of transformations, nothing will be executed. Each time an action is called on a Pipeline all transformations will be recomputed. To prevent this, parts of the pipeline can be cached by calling the appropriate methods (cache or persist).[3]<\/p>\n
Assuming we continue the code example above and we want to invoke operations on the firstRDD object. In the code below, on the RDD object a method map is called, which is a higher order function. This method we pass a Java Lambda expression. The map function transforms each string in the list into upper-case letters. The reduce function concatenates the strings pairwise. The result is one string with all strings of the RDD transformed into uppercase-letters and concatenated together.<\/p>\n
String sentence = firstRDD.map( str -> str.toUpperCase())\n .reduce( (str1,str2) -> str1 +\" \"+str2);\n\/\/ result: THIS IS A FIRST SPARK TEST<\/pre>\nRDD Persistence – Storage Levels<\/h4>\n
Per default Spark caches all intermediate results, like the result of the map operation in the example above, in memory. The storage level can be configured. For example if data with huge size is processed the intermediate results can be stored on disk. The possible storage levels can be viewed in the programming guide of Apache Spark: http:\/\/spark.apache.org\/docs\/latest\/programming-guide.html#rdd-persistence<\/a>.<\/p>\n
So at this point we covered the very basics of Apache Spark. There are more concepts and operations which you can read on the Apache Spark programming guide<\/a>.<\/p>\n
Spark SQL<\/h4>\n
SparkSQL is something similar to Apache Hive. Both allow querying structured data from heterogeneous data sources with SQL statements.<\/p>\n
In the example below the json method of an Spark SQL Context loads a JSON-file containing multiple people-JSON objects and transforms the jsons into rows of a data frame.<\/p>\n
\/\/ create sql context with java spark context\nSQLContext sqlContext = new SQLContext(jsc);\n\/\/ read people.json and transform into data frame\nDataFrame dfPeople = sqlContext.read().json(\"hdfs:\/\/path\/to\/file\/people.json\");<\/pre>\nA data frame in Spark can be compared to relational database tables or a python data frame [6]. That means the JSON-file above is transformed into a table where the attributes represents the columns like pictured below. A people JSON simply consists of the two attributes name and age.<\/p>\n
![](\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/08\/streaming-flow.png\")
<\/p>\n