How to deploy tests with Maven on a Virtual Machine via Git

How to deploy tests with Maven on a Virtual machine.

In this tutorial you will learn how to deploy your Java project onto any VM and how to deploy and run it via MVN.

We will go over the following points.

1. Add project to github

We first need to have a github project

2.Add SSH keys on VM (optional)

Add a SSH key on your VM and then add it to your github account. Follow this tutorial if you are not sure how to do this : https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/

3. Clone Project to VM

Get the URL of your repo and clone it to your VM

4. Compile project

mvn clean test-compile

5. Run Tests

mvn test

This will execute all you compiled tests within the /target/test-classes folder.

6. Run a specific test function

mvn -Dtest=testClass#testMethod test

This command will run the method “testMethod” inside of the “testClass”

Recap – What have we learned?

Congratulations, you have learned how to elegantly deploy your Java project on a remote virtual machine using Git and Maven.

Basic Kafka Producer, How to write records in Java

Basic Kafka Producer, How to write records in Java

You will learn

  • How to configure Kafka Consumer properties
  • How to create Kafka consumer
  • How to send records with Kafka Consumer

The first example shows how to print out records from Kafka to the console.

We will have to set the properties for a Kafka Consumer Object and create it.

Then we subscribe to the topics of our choice

Setting Properties

Properties props = new Properties();
props.put(“bootstrap.servers”, “localhost:9092”);
props.put(“acks”, “all”);
props.put(“retries”, 0);
props.put(“batch.size”, 16384);
props.put(“linger.ms”, 1);
props.put(“buffer.memory”, 33554432);
props.put(“key.serializer”, “org.apache.kafka.common.serialization.StringSerializer”);
props.put(“value.serializer”, “org.apache.kafka.common.serialization.StringSerializer”);

Create an Instance of a Kafka Producer, with the properties we just set

Producer producer = new KafkaProducer<>(props);

Cal the send() function on the producer object to send a record to the Kafka topic wetter

for (int i = 0; i < 100; i++) { producer.send(new ProducerRecord(“wetter”, Integer.toString(x), Integer.toString(i)));

}

Basic Kafka Consumer Java example (How to print Kafka records to terminal)

Basic Kafka Consumer example in java

In this tutorial you will learn, how to make a simple kafka consumer, which sends records for ever.

You will learn the following :

  • How to create topics
  • How to create Kafka Consumer
  • How to configure Kafka Properties
  • How to subscribe to a topic
  • How to poll infinitely with a consumer

You will learn how to create topics, how to create a Kafka consumer and how to configure it.

Also, you w

The first example shows how to print out records from Kafka to the console.

We will have to set the properties for a Kafka Consumer Object and create it.

Then we subscribe to the topics of our choice

Setting Properties

Java

Properties props = new Properties();
props.put(“bootstrap.servers”, “localhost:9092”);
props.put(“group.id”, “test”);
props.put(“enable.auto.commit”, “true”);
props.put(“auto.commit.interval.ms”, “1000”);
props.put(“session.timeout.ms”, “30000”);
props.put(“key.deserializer”, “org.apache.kafka.common.serialization.StringDeserializer”);
props.put(“value.deserializer”, “org.apache.kafka.common.serialization.StringDeserializer”);

Create an Instance of a KafkaConsumer Object and call the subscribe() method to subscribe to some topics

KafkaConsumer consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList(“wetter”, “bar”));

Ask the consumer for new records continuously

while (true) {
ConsumerRecords records = consumer.poll(100);
for (ConsumerRecord record : records) {
System.out.printf(“offset = %d, key = %s, value = %s”, record.offset(), record.key(), record.value());
}
}

Deployment Cylcle with Spark and Hadoop with java

This article will show you one of many possible cycles to deploy your code as quickly and efficiently as possible. Also, we will talk a little about what Hadoop and Spark actually is and how we can use it to make awesome distributed computations!

What is Hadoop and Spark for?

You use your Spark cloud, to do very computationally expensive tasks in a distributed fashion.

Hadoop provided the data in a distributed fashion, making it available from multiple nodes and by that increasing the rate at which every node in the cluster network will get its data.

We will write our code in Java and define cluster computations using the open source Apache Spark library.

After defining the code, we will use Maven to create a fat jar from it, which will contain all the dependencies.

We will make the Jar available from multiple sources, so that multiple computation nodes from our spark cluster can download it at the same time, this is achieved by making the data available distributed through hadoop.

What does a deployment cycle with kafka and hadoop look like in Java?

A typical cycle could look like this :

  1. Write code in Java
  2. Compile code into a fat Jar
  3. Make jar available in Hadoop cloud
  4. Launch Spark Driver which can allocate a dynamic amount of nodes to take care of the computations defined within the jar.

1.Write code in Java

You will have to define a main function with a main class. This will be the code that the cluster runs first, so everything starts from this function.

2. Compile code into fat Jar

mvn clean compile assembly:single

3. Make jar available from Hadoop cloud

Go into your Hadoop web interface and browse the file system

3.1 Create a folder in the cloud and upload the jar

After uploading your jar into your Hadoop cloud, it will be available to any computer that can talk with the Hadoop cloud. It is now distributed available on all the Hadoop nodes and is ready for highly efficient and fast data exchange with any cluster, in our example we use a Spark cluster.

If your hadoop node is called hadoop_fs and port is 9000, your jar is available to any node under the following URL:

hdfs://hadop_fs:9000/jars/example.jar

4. Launch distributed Spark Computation

To launch the driver, you need an instance of the spark-submit class. The most straightforward way to get it, is to just download the Spark library and unzip it.

wget http://apache.lauf-forum.at/spark/spark-2.3.2/spark-2.3.2-bin-hadoop2.7.tgz

4.1 Launch Spark driver from command line.

Go to the directory where you have unzipped your Spark library, for me it would be

loan@Y510P:~/Libraries/Apache/Spark/spark-2.3.0-bin-hadoop2.7$

In the ./bin/spark-submit will have all the functionality we will require,

4.2 Gathering the Parameters

You need the following parameters, to launch your jar in the cluster

  • Spark Master URL
  • Hadoop Jar Url
  • Name of your main Class
  • Define –deploy-mode as Cluster to run the computation in cluster mode

4.3 Final step :Put the parameters together and launch the Jar in the cluster

./bin/spark-submit –class com.package.name.mainClass –master spark://10.0.1.10:6066 –deploy-mode cluster hdfs://hadop_fs:9000/jars/example.jar

This will tell the Spark cluster, where the Jar we want to run is. It will launch a user defined (or appropriate) amount of executors and finish the computation in a distributed fashion

Your Task should now show up in your Spark Webinterface.

What have you learned :

  • How to turn your java code into a fat jar
  • How to deploy your fat jar into the Hadoop cloud
  • How to run your code distributed in Spark, usinsg Hadoop as data source