Laws - or principles - can give us guidance and teach us lessons from our peers’ mistakes. In this article, I will introduce you to five laws I always have in the back of my mind when designing or implementing a software. Some of them relate to pure development, some

MongoDB and Apache Spark are two popular Big Data technologies. In my previous post, I listed the capabilities of the MongoDB connector for Spark. In this tutorial, I will show you how to configure Spark to connect to MongoDB, load data, and write queries. To demonstrate how to use Spark

MongoDB is one of the most popular NoSQL databases. Its unique capabilities to store document-oriented data using the built-in sharding and replication features provide horizontal scalability as well as high availability. Apache Spark is another popular “Big Data” technology. Spark provides a lower entry level to the world of distributed

JHipster is an open source project that generates a fully working application in seconds. With a minimal configuration, JHipster accelerates the start of new projects by integrating frontend, backend, security and a database. Cassandra is one of the supported databases and JHipster generates all the configuration needed to access the

SBT is the favorite build tool for Scala developers, it is used by the Play framework and by the Activator module from Typesafe. SBT is easy to get started with, offers a wide range of features and is extremely easy to extend. It is the perfect mix between Maven and

With the recent announcement of the Community Edition, it’s time to have a look at the Databricks Cloud solution. Databricks Cloud is a hosted Spark service from Databricks, the team behind Spark. Databricks Cloud offers many features: A cluster management service. The service will spin up Amazon EC2 instances

In a previous post, we’ve seen why it’s important to test your Spark jobs and how you could easily unit test the job’s logic, first by designing your code to be testable and then by writing unit tests. In this post, we will look at applying the

Like any other application, Apache Spark jobs deserve good testing practices and coverage. Indeed, the costs of running jobs with production data makes unit testing a must-do to have a fast feedback loop and discover the errors earlier. But because of its distributed nature and the RDD abstraction on top