Spark Error ” Exception in thread “main” java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running” MetricsSystem”

I recently did some feature engineering on a few datasets with spark.

I wanted to make the datasets available in our Hadoop cluster, so i used our normal dataset upload pattern, but ran into these  nasty little errors

 

 

and also

 

 

 

The full exception stacks looks like this

 

 

So what do these error mean and why do they occur?

  • It often has to do something, with not initializing the Spark session properly
  • Usually means, there is a wrong value, for the master location
  • Double check the master adress for the spark session, by default it should use port 7077 and NOT  6066
  • Check if the version of the spark cluster  is the same as your spark version in the jar / of the job you want to submit

How do I fix “Could not find CoarseGrained Scheduler” or “an only call  getServletHandlers on a running MetricsSystem”

  • Update master URL
  • Update dependencies POM/Gradle/Jar, so you use the same version as the cluster

Now your error should be fixed. Have fun with your Spark Cluster!

 

Convolution Operators

Convolution Operators

Overview of  a few convolution operators

Sobel Operator for Edge Detection

The Sobel Operator calculates the pixels first order derivatives in the X and Y direction.

If the is little derivative, there is no unstructured area

If the derivative high in one direction, we have an edge in that direction

If the derivative high in two directions  , we have a corner / point of interest

 Gaussian Smoothing

Replace each pixel by it’s local average

.It is used for scale Space

The Laplace  Operator for

The differnce compard to the Sobel operator is, that it uses the second order derrivative.

This makes the Laplace operator very sensitive to noisy.

Edges are, where the second derivative is crossing over to 0 ( Zweite ableitung = 0 Hochpunkt!)

Laplacian of Gaussian Filter :

  1.  First smooth with gaussian filter.

  2. Then find zero crossings with the Laplacian filter.

  3. Can also combine one into one LOG  convolution

Double of Gaussian

LoG does not have to be calculated, it can be also approximated by calculating the difference between two Gaussian Filters at different scales . DoG

SIFT Detector

  1. Multiple DoG filters are applied to the image at different scales
  2. The resulting images are stacked on top each other to create a 3D volume
  3. Points that are local extrema inside of the 3D voluma, are considerd points of interest.  
  4. Remove bad points, like candidates in smooth regions  or directly on top of edges

SIFT Discriptor

  1. For a detected point of interest, choose 16×16 region around the point
  2. Compute each gradient for each pixel
  3. Subdivide into 16 * 4×4 groups
  4. Compute orientation histogram
  5. Glue the histograms together to get a 128 element feature vector

Characteristics

  • SIFT is very resillient to changes (invariant)   to constant intensity changes, as itt is based on gradients
  • Very invariant to contreast changes  , as the histogram binning provides normalization
  • invarient to small deformations  

Scale invariance is caused by the SIFT detector

How to fix Java Error “The trustAnchors parameter must be non-empty” while building Maven Single Jar

Recently I tried to deploy my Java  project as a single fat jar. To my bad awekening, Maven did not feel like creating jars anymore and complained with errors like :

You might encounter an error like this, if you recently reinstalled your java and want to build Jar with maven

Or

Or

Or

Or

Or

 

Or

 

Or

 

The console with spit something like this out :

 

 

 

How to fix the error

To fix the error, we will download the java JDK and extract the certificates into our local java installation.

Get java

 

Extract the jdk

 

 

Copy the Certificate into your java directory

 

And now you can finally build your jar with