spark-notes

This project is maintained by spoddutur

Apache Ignite to hold data that needs rebroadcasting

Continuation from part-3..

Earlier Spark-Native solutions are more-or-less compromised work-arounds:

We’ve discussed in part-2 and part-3 different spark-native approaches to maintain a mutable distributed cache native-within-spark such as:

  1. Hold the cache as globalCorpus at the driver which gets updated once per batch aggregating the localCorpus learnt from the current batch received.
  2. Hold this cache as broadcast variable and Re-broadcast it periodically as it keeps changing.

But, both these approaches seem more or less like compromised work-arounds given the fact that RDD’s in spark are immutable. Let me elaborate more on this:

ApacheIgnite - A non-spark-native solution:

Yes! ApacheIgnite is an in-memory distributed SQL DB which comes with database caching as well.

Following figure illustrates the role of ApacheIgnite within our Spark application: image

Theory:

Ignite data can be exposed as mutable IgniteRDD in Spark which has benefits such as:

How does Updates/Ingestion happen to IgniteRDD: Ignite Streamers are streaming components, that ingest data in the fastest way possible into apache ignite. Hence, any streaming real-time cache-data updates/inserts happens fast.

Demo:

Here, I’ll demo how to mine phrases in a spark streaming application and keep track of changing cache in ignite. Note that, we saw solution for the same in part2 without using any external service.

val sentencesDf = spark.read
   		       .format("text")
		       .load(“/tmp/gensim-input”).as[String]

val igniteWordsRDD = new IgniteContext[String, Int](sc, () => new IgniteConfiguration()).fromCache(“igniteWordsRDD")

// learn phrases from input
val phrasesRdd = sentencesDf.flatMap(sentence => Phrases.learnVocab(sentence))

// saving the phrase counts to ignite cache
igniteWordsRDD.saveValues(phrasesRdd)

// transform values in ignite cache - merge old phraseRecords and new phraseRecords
igniteWordsRDD.groupBy(“phrase”).agg(sum($”count”))

Conclusion: Analysis and Advantages:

Key Takeouts:

  1. Now, hopefully, you would agree with me and stop thinking of modifying a broadcast variable.
  2. Categorize your need to broadcast into
    1. Only collecting cache data as we see more and more batches
    2. Collect cache data nd also use the latest cached data within our application.
  3. We saw how to solve solutions to both the above scenarios without relying any external service. It does computation in a distributed fashion and collects changes at one place i.e., driver.
  4. Spark-native approaches are simple for cases where cache-size is small or its some quick POC todo
  5. Else, as we have seen in this article, using an external service like ApacheIgnite eases user off quite some payload and brings on table some other benefits like seamless weaving with RDD’s and cross-application cache sharing.
  6. With Ignite, we saw how both computation and storage is happening in a distributed fashion.

References