---
title: Intro to MapReduce operations with Spark
description: "In the previous post, we used the Map operation which allows us to transform\nvalues with a transformation function. We will now explore the Reduce operation\nwhich produces aggregates. Thus, we will work in MapReduce just like we do with\nHadoop.\n\nTheory\nWith Spark, just like Hadoop, a Reduce operation"
image: https://blog.ippon.tech/hubfs/Imported_Blog_Media/ippon-bkr-Jan-18-2024-06-04-17-4565-PM.png
---
[Skip to main content](https://blog.ippon.tech/intro-mapreduce-spark#main)
[   ](https://ipponusa.com/)
- [Home](https://ipponusa.com)
- [Show submenu for About About](https://ipponusa.com/about-us)
- [Who We Are](https://ipponusa.com/who-we-are)
- [Careers](https://ipponusa.com/careers)
- [Show submenu for Our Partners Our Partners](https://ipponusa.com/our-partners/)
- [AWS](https://ipponusa.com/our-partners/aws/)
- [Snowflake](https://ipponusa.com/our-partners/snowflake/)
- [Databricks](https://ipponusa.com/our-partners/databricks/)
- [Microsoft](https://ipponusa.com/our-partners/microsoft/)
- [Show submenu for Services Services](https://ipponusa.com/services)
- [Snowflake](https://ipponusa.com/service/snowflake-concierge/)
- [Artificial Intelligence](https://ipponusa.com/service/artificial-intelligence/)
- [Data & Analytics](https://ipponusa.com/service/data-analytics/)
- [Cloud Strategy](https://ipponusa.com/service/cloud-strategy/)
- [Platform Modernization](https://ipponusa.com/service/platform-moderization/)
- [Product Innovation](https://ipponusa.com/service/product-innovation/)
- [Operating Model](https://ipponusa.com/service/operating-model/)
- [Blogs](https://blog.ippon.tech/)
- [Success Stories](https://ipponusa.com/success-stories/)
- [Show submenu for Resources Resources](https://ipponusa.com/resources/)
- [eBooks](https://ipponusa.com/ebooks/)
- [The Data Pour](https://info.ippon.tech/the-data-pour)
- [Videos & Webinars](https://info.ippon.tech/videos-and-webinars)
- [Media Center](https://ipponusa.com/media-center/)
Search
Open main navigation
Close main navigation
- [Home](https://ipponusa.com)
- Show submenu for About About
- About
- [About](https://ipponusa.com/about-us)
- [Who We Are](https://ipponusa.com/who-we-are)
- [Careers](https://ipponusa.com/careers)
- Show submenu for Our Partners Our Partners
- Our Partners
- [Our Partners](https://ipponusa.com/our-partners/)
- [AWS](https://ipponusa.com/our-partners/aws/)
- [Snowflake](https://ipponusa.com/our-partners/snowflake/)
- [Databricks](https://ipponusa.com/our-partners/databricks/)
- [Microsoft](https://ipponusa.com/our-partners/microsoft/)
- Show submenu for Services Services
- Services
- [Services](https://ipponusa.com/services)
- [Snowflake](https://ipponusa.com/service/snowflake-concierge/)
- [Artificial Intelligence](https://ipponusa.com/service/artificial-intelligence/)
- [Data & Analytics](https://ipponusa.com/service/data-analytics/)
- [Cloud Strategy](https://ipponusa.com/service/cloud-strategy/)
- [Platform Modernization](https://ipponusa.com/service/platform-moderization/)
- [Product Innovation](https://ipponusa.com/service/product-innovation/)
- [Operating Model](https://ipponusa.com/service/operating-model/)
- [Blogs](https://blog.ippon.tech/)
- [Success Stories](https://ipponusa.com/success-stories/)
- Show submenu for Resources Resources
- Resources
- [Resources](https://ipponusa.com/resources/)
- [eBooks](https://ipponusa.com/ebooks/)
- [The Data Pour](https://info.ippon.tech/the-data-pour)
- [Videos & Webinars](https://info.ippon.tech/videos-and-webinars)
- [Media Center](https://ipponusa.com/media-center/)
- Search
- [Contact Us](https://ipponusa.com/contact/)
[Contact Us](https://ipponusa.com/contact/)
# Intro to MapReduce operations with Spark
by [Alexis Seigneurin](https://blog.ippon.tech/author/alexis-seigneurin)
November 22, 2014
Nov 22, 2014
In the previous post, we used the Map operation which allows us to transform values with a transformation function. We will now explore the Reduce operation which produces aggregates. Thus, we will work in MapReduce just like we do with Hadoop.
## Theory
With Spark, just like Hadoop, a Reduce operation is an operation that aggregates values two by two, by using as many steps as necessary to process all the elements of the collection. This is what allows the framework to perform aggregations in parallel, on multiple nodes when needed. The framework will choose two elements and pass them to a function we will define. The function must return the new element that will replace the two first. This means that the output type must be identical to the input type: values must be homogeneous, so that the operation can be repeated until all elements are processed. With Spark there are two reduction operations:
- **reduce()** works on elements, whatever their type, and returns a unique value.
- **reduceByKey()** works on values associated to the same key. This operation is only possible on RDDs of type JavaPairRDD (a list of key-value tuples), and it produces a result which is a JavaPairRDD too, but in which each key appears only once (equivalent to a key-value Map).
We are going to study the reduce() operation in this blog post. The reduceByKey() operation will be addressed in an upcoming article.
## Value aggregation in MapReduce
In our first code example, we built an RDD containing the heights of trees in Paris. We used the count() operation to count trees with a certain height.
```
long count = sc.textFile(“arbresalignementparis2010.csv”)
.filter(line -> !line.startsWith(“geom”))
.map(line -> line.split(“;”))
.map(fields -> Float.parseFloat(fields[3]))
.filter(height -> height > 0) .count();
```
Instead of just counting non-null heights, we can use the reduce() operation to calculate the total height of trees. To do that, we need a function that will takes two heights as input and returns their sum. This function has two parameters of type Float and returns a Float:
This function can be written as a lambda expression:
```
(x, y) -> x + y
```
Thus, we can write:
```
float totalHeight = sc.textFile(“arbresalignementparis2010.csv”)
…
.reduce((x, y) -> x + y);
```
The framework will call our reduce function until every value is processed.
## Counting in Map Reduce
The count() operation we previously saw can also be written as a reduction operation. Since the expected result is an amount of trees (int or Integer), we first have to convert input elements (Float type) to have consistent data. We will then proceed in two steps:
- a **map()** operation to convert each element to the value 1
- a **reduce()** operation to add up the 1s
```
long count = sc.textFile(“data/arbresalignementparis2010.csv”)
…
.map(item -> 1)
.reduce((x, y) -> x + y);
```
Note that the lambda expression used in the reduce() method is identical to the one used we used previously, but the equivalent function manipulates Integer objects.
## Calculating the average tree height
We can calculate the average tree height in our file by doing:
- an aggregation operation (sum) of heights
- a count of elements
- a division of the first value by the second
Since the aggregation and the counting are based on the same file and the first operations are identical, we can reuse the same RDD. We will then use the cache() operation which allows us to cache the RDD in memory. The calculations will thus be executed only once and the intermediary result will be directly used for both operations of aggregation and counting.
```
JavaRDD rdd = sc.textFile(“data/arbresalignementparis2010.csv”)
.filter(line -> !line.startsWith(“geom”))
.map(line -> line.split(“;”))
.map(fields -> Float.parseFloat(fields[3]))
.filter(height -> height > 0)
.cache();
float totalHeight = rdd.reduce((x, y) -> x + y);
long count = rdd.count();
System.out.println(“Count: ” + count);
System.out.println(“Total height: ” + totalHeight);
System.out.println(“Average height ” + totalHeight / count);
```
We get the following result:
```
Count: 32112
Total height: 359206.0
Average height 11.186036
```
## Conclusion
Spark performs aggregations on all values through a Reduce operation that processes elements two by two, in order to distribute the processing. In the next post, we will see how to perform aggregation operations by key. You can find the code example on [Github](https://github.com/aseigneurin/spark-sandbox?ref=blog.ippon.tech).
## Related Articles
### Comments
### Subscribe to Our Blog!
Stay informed with the latest insights and updates by signing up for our weekly blog newsletter – delivered straight to your inbox!
[Back to blog homepage »](https://blog.ippon.tech/)
Ippon is a consulting and expertise firm, who is convinced that technology is a source of progress for society. We help our clients leverage their digital assets to design an appropriate strategy and deploy their transformation roadmap at scale.
#### Navigation
- [Home](https://ipponusa.com/)
- [About](https://ipponusa.com/about-us/)
- [Services](https://ipponusa.com/services/)
- [Blogs](https://blog.ippon.tech/?__hstc=223043268.1036d82247623b329dfefe4e92697801.1714729179243.1714729179243.1714729179243.1&__hssc=223043268.2.1714729179244&__hsfp=803678701)
- [eBooks](https://ipponusa.com/ebooks/)
- [The Data Pour](https://info.ippon.tech/the-data-pour?__hstc=223043268.1036d82247623b329dfefe4e92697801.1714729179243.1714729179243.1714729179243.1&__hssc=223043268.2.1714729179244&__hsfp=803678701)
[Contact Us](https://ipponusa.com/contact/)
[Join Us](https://ipponusa.com/careers/)
#### Services
- [Data & Analytics](https://ipponusa.com/service/data-analytics/)
- [Cloud Strategy](https://ipponusa.com/service/cloud-strategy/)
- [Artificial Intelligence](https://ipponusa.com/service/artificial-intelligence/)
- [Operating Model](https://ipponusa.com/service/operating-model/)
- [Platform Modernization](https://ipponusa.com/service/platform-moderization/)
- [Product Innovation](https://ipponusa.com/service/product-innovation/)
#### Ippon International
- [France](https://fr.ippon.tech/?__hstc=223043268.1036d82247623b329dfefe4e92697801.1714729179243.1714729179243.1714729179243.1&__hssc=223043268.2.1714729179244&__hsfp=803678701)
- [Australia](https://au.ippon.tech/?__hstc=223043268.1036d82247623b329dfefe4e92697801.1714729179243.1714729179243.1714729179243.1&__hssc=223043268.2.1714729179244&__hsfp=803678701)
- #### Contact
- Ippon Technologies
[3431 West Leigh Street Richmond, VA 23230, USA](https://maps.app.goo.gl/RySm6SAwZnsxeiLz9)
- [(844) 477- 6687](tel:8444776687)
- [Sales@ipponusa.com](mailto:Sales@ipponusa.com)
©Copyright 2024 Ippon USA. All Rights Reserved. | [Terms and Conditions](https://ipponusa.com/privacy-policy/) | [Privacy Policy](https://ipponusa.com/privacy-policy/) | [Website by Skol Marketing](https://skolmarketing.com/)
```json
{
"@context" : "https://schema.org",
"@type" : "BlogPosting",
"author" : {
"@type" : "Person",
"name" : "Alexis Seigneurin",
"url" : "https://blog.ippon.tech/author/alexis-seigneurin"
},
"dateModified" : "2024-01-29T17:12:32.574Z",
"datePublished" : "2014-11-22T14:23:00.000Z",
"headline" : "Intro to MapReduce operations with Spark",
"image" : [ "https://blog.ippon.tech/hubfs/Imported_Blog_Media/ippon-bkr-Jan-18-2024-06-04-17-4565-PM.png" ],
"mainEntityOfPage" : {
"@id" : "https://blog.ippon.tech/intro-mapreduce-spark",
"@type" : "WebPage"
},
"publisher" : {
"@type" : "Organization",
"logo" : {
"@type" : "ImageObject",
"url" : "https://blog.ippon.tech/hubfs/logo_SVG.svg"
},
"name" : "Ippon Technologies"
}
}
```