All posts by loan

Spark Kubernetes error Can only call getServletHandlers on a running MetricsSystem – How to fix it

Did you encounter an error like

java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem

The fix is pretty straight forward.
The error is caused by running a different Spark version in the cluster then the one used for Spark submit.

Double-check the cluster Spark version via for the cluster UI or checking the Spark Master pod.
Then check the version of your spark-submit, it is usually shown during submitting a job.

This is the full stack trace :

java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:224)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.(SparkContext.scala:516)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at phabmacsJobs.KafkaJsonWriter$.main(KafkaJsonWriter.scala:32)
at phabmacsJobs.KafkaJsonWriter.main(KafkaJsonWriter.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:224)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.(SparkContext.scala:516)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at phabmacsJobs.KafkaJsonWriter$.main(KafkaJsonWriter.scala:32)
at phabmacsJobs.KafkaJsonWriter.main(KafkaJsonWriter.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Tensorflow 1.x and 2.x Saving Error Using a `tf.Tensor` as a Python `bool` is not allowed.

Ever tried to save a Tensorflow model with tf.compat.v1.saved_model.simple_save or a similar TF saving method function?

Ever Encounterd in TF 1,X
TypeError: Using a tf.Tensor as a Python bool is not allowed. Use if t is not None: instead of if t: to test if a tensor is defined, and use TensorFlow ops such as tf.cond to execute subgraphs conditioned on the value of a tensor.
or in TF 2.X
OperatorNotAllowedInGraphError: using a tf.Tensor as a Python bool is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.

Error
DONT BE FOOLED! This error is not what it seems. At least for me…

This was my save method that was causing the error


token_tensor = tf.ones((input_len,batch_size), "int32", "token_tensor")
segment_tensor = tf.ones((input_len,batch_size), "int32", "segment_tensor")
mask_tensor = tf.ones((input_len,batch_size), "float32", "mask_tensor")
seq_out = model.get_sequence_output()

with tf.compat.v1.Session() as sess:
tf.compat.v1.saved_model.simple_save(
sess,
export_dir,
inputs={'input': token_tensor, 'segment' : segment_tensor, 'mask' : mask_tensor},
outputs=seq_out,
legacy_init_op=init_op
)

See the error? Its very minor…
The problem was, the output tensor IS NOT INSIDE OF A DICT !
Duuuuh! Isn’t that obvious to infer from the

Looking at the source code of the save function is what actually made me see the issue!
simple_save.py

So here is the fix, just define a dict for your input or output tensors!

token_tensor = tf.ones((input_len,batch_size), "int32", "token_tensor")
segment_tensor = tf.ones((input_len,batch_size), "int32", "segment_tensor")
mask_tensor = tf.ones((input_len,batch_size), "float32", "mask_tensor")
seq_out = model.get_sequence_output()

with tf.compat.v1.Session() as sess:
tf.compat.v1.saved_model.simple_save(
sess,
export_dir,
inputs={'input': token_tensor, 'segment' : segment_tensor, 'mask' : mask_tensor},
outputs={"out": seq_out},
legacy_init_op=init_op
)

Happy TensorFlow hacking!

This is the full Stack trace in TF 1.X
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
in ()
45 outputs= seq_out, #{'output': mask_tensor, 'norms': mask_tensor},
46 #outputs={'word_emb': model_wordembedding_output, 'sentence_emb': model_sentence_embedding_output},
---> 47 legacy_init_op=init_op
48 )
49 # print('saving done')

/home/loan/venv/XLNET_jupyter_venv/lib/python2.7/site-packages/tensorflow/python/util/deprecation.pyc in new_func(*args, **kwargs)
322 'in a future version' if date is None else ('after %s' % date),
323 instructions)
--> 324 return func(*args, **kwargs)
325 return tf_decorator.make_decorator(
326 func, new_func, 'deprecated',

/home/loan/venv/XLNET_jupyter_venv/lib/python2.7/site-packages/tensorflow/python/saved_model/simple_save.pyc in simple_save(session, export_dir, inputs, outputs, legacy_init_op)
79 signature_def_map = {
80 signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:
---> 81 signature_def_utils.predict_signature_def(inputs, outputs)
82 }
83 b = builder.SavedModelBuilder(export_dir)

/home/loan/venv/XLNET_jupyter_venv/lib/python2.7/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.pyc in predict_signature_def(inputs, outputs)
195 if inputs is None or not inputs:
196 raise ValueError('Prediction inputs cannot be None or empty.')
--> 197 if outputs is None or not outputs:
198 raise ValueError('Prediction outputs cannot be None or empty.')
199

/home/loan/venv/XLNET_jupyter_venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.pyc in __nonzero__(self)
702 TypeError.
703 """
--> 704 raise TypeError("Using a tf.Tensor as a Python bool is not allowed. "
705 "Use if t is not None: instead of if t: to test if a "
706 "tensor is defined, and use TensorFlow ops such as "

TypeError: Using a tf.Tensor as a Python bool is not allowed. Use if t is not None: instead of if t: to test if a tensor is defined, and use TensorFlow ops such as tf.cond to execute subgraphs conditioned on the value of a tensor.

And Tensorflow 2.x

---------------------------------------------------------------------------
OperatorNotAllowedInGraphError Traceback (most recent call last)
in
86 inputs=bert_inputs,
87 outputs=table_tensor,
---> 88 legacy_init_op=init_op
89 )
90

~/venv/XLNET_py3_venv/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py in new_func(*args, **kwargs)
322 'in a future version' if date is None else ('after %s' % date),
323 instructions)
--> 324 return func(*args, **kwargs)
325 return tf_decorator.make_decorator(
326 func, new_func, 'deprecated',

~/venv/XLNET_py3_venv/lib/python3.7/site-packages/tensorflow_core/python/saved_model/simple_save.py in simple_save(session, export_dir, inputs, outputs, legacy_init_op)
79 signature_def_map = {
80 signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:
---> 81 signature_def_utils.predict_signature_def(inputs, outputs)
82 }
83 b = builder.SavedModelBuilder(export_dir)

~/venv/XLNET_py3_venv/lib/python3.7/site-packages/tensorflow_core/python/saved_model/signature_def_utils_impl.py in predict_signature_def(inputs, outputs)
195 if inputs is None or not inputs:
196 raise ValueError('Prediction inputs cannot be None or empty.')
--> 197 if outputs is None or not outputs:
198 raise ValueError('Prediction outputs cannot be None or empty.')
199

~/venv/XLNET_py3_venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py in __bool__(self)
755 TypeError.
756 """
--> 757 self._disallow_bool_casting()
758
759 def __nonzero__(self):

~/venv/XLNET_py3_venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py in _disallow_bool_casting(self)
524 else:
525 # Default: V1-style Graph execution.
--> 526 self._disallow_in_graph_mode("using a tf.Tensor as a Python bool")
527
528 def _disallow_iteration(self):

~/venv/XLNET_py3_venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py in _disallow_in_graph_mode(self, task)
513 raise errors.OperatorNotAllowedInGraphError(
514 "{} is not allowed in Graph execution. Use Eager execution or decorate"
--> 515 " this function with @tf.function.".format(task))
516
517 def _disallow_bool_casting(self):

OperatorNotAllowedInGraphError: using a tf.Tensor as a Python bool is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.

Kubernetes Helm Install Error: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

Error: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

If you ever run into this error message when installing a chart with Helm into Kubernetes, try closing your Kubernetes connection and open it again!
In Azure AKS it would be
az aks browse --resource-group bachelor-ckl --name aks-ckl

This resolves the Error Kubernetes Helm Install Error: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request.

Happy Devops -ing!

Creating a crontab with just 1 command

In this tutorial you will learn, how to add a cronjob with just one command to your crontab file.
This makes deploying virtual machines and their automatic configuration of Cronjobs easy!

That is all you need! This command will be appended to your crontab files.

 

It is pretty handy for setting up cronjobs on fresh Ubuntu VMs without requiring root.

Happy Deploying!

Creating Cromjobs

What is a cronjob or a crontab file?

Crontab (cron table) is a text file that specifies the schedule of cron jobs. There are two types of crontab files. The system-wide crontab files and individual user crontab files.

Users crontab files are stored by the user’s name and their location varies by operating systems. In Red Hat based system such as CentOS, crontab files are stored in the /var/spool/cron directory while on Debian and Ubuntu files are stored in the /var/spool/cron/crontabs directory.

Although you can edit the user crontab files manually, it is recommended to use the crontab command.

/etc/crontab and the files inside the /etc/cron.d directory are system-wide crontab files which can be edited only by the system administrators.
In most Linux distributions you can also put scripts inside the /etc/cron.{hourly,daily,weekly,monthly} directories and the scripts will be executed every hour/day/week/month

Linux Crontab Command

The crontab command allows you to install or open a crontab file for editing. You can use the crontab command to view, add, remove or modify cron jobs using the following options:

  • crontab -e – Edit crontab file, or create one if it doesn’t already exist.
  • crontab -l – Display crontab file contents.
  • crontab -r – Remove your current crontab file.
  • crontab -i – Remove your current crontab file with a prompt before removal.
  • crontab -u – Edit other use crontab file. Requires system administrator privileges.

Docker glossary

Terminology

  • Container Image
    • All the dependencies, deployment and execution configuration and needed to create a container packed together. Usually, an image is derived from multiple base images that are layer stacked on top of each other, to form the containers filesystem. Images are immutable once created.
  • Dockerfile
    • Text file that defines how to build a Docker image. Like a batch script, defining which images to use, which programs to install and which files to copy to get the environment working as needed.
  • Build
    • The act of building an container image based on the data provided by the Dockerfile and the other files in the same folder as the image creation folder.
  • Container
    • One instantiation of a Docker image. It represents a single process/ application/ service, running on the images host. It contains the Docker image, an execution environment and a standard set of instructions. For scaling to millions of users, you just have to deploy more containers and balance the work across them, so easy !
  • Volumes
    • Since images are immutable, a containerized application cannot write to it’s own image! That is, why we need volumes. They are an extra layer managed by Docker, which emulated a filesystem on the docker host machine, to which the containerized application will write. The containerized application does not notice the difference and act’s like usual, when working with the volumes.
  • Tag
    • A label or identifier which can be applied to an images, so multiple different versions of the same image can be identified.
  • Multi-stage Build
    • Use a large base image for compiling and publishing the application and then use the publishing folder with a small runtime-only base image, to produce a much smaller final image.
  • Repository
    • A collection of Docker images
  • Registry
    • A service, to access a Repository
  • Multi-arch image
    • For multi-architectures (Windows, Linux) it is a features, that automatically requests the proper image, when running a Dockerfile.
  • Docker-Hub
    • Public registry to upload images and work with them. Build triggers, web hooks, integration to GitHub and Bitbucket
  • Azure Container Registry
    • A registry for Docker images and its components in Azure
  • Docker Trusted Registry
    • a registry server for Docker that can be hosted on your own private server for private images
  • Docker Community Edition (CE)
    • 12
  • Docker Enterprise Edition (EE)
    • Enterprise-scale version of Docker tools for Linux and Windows development
  • Compose
    • Command-line tool and YAML file format for defining and running multi-container applications. You can define a single application based on multiple images with one or more YAML files and then deploy all of the containers/images with just one command.
    • Imagine, first set of images setup Hadoop, the next images setup Kafka, the next Spark and the last sets up some Java web server
  • Cluster
    • A collection of Docker hosts exposed as if they were a single virtual Docker host, so basically put many versions of the same container behind the same IP, to handle huge amounts of users aka to handle scaling.
    • Docker clusters can be created with Kubernetes, Azure Service Fabric, Docker Swarm and Mesosphere DC/OS
  • Orchestrator
    • Tool to simplify management of cluster and Docker hosts. Orchestrators enable you to manage their images, containers and hosts through a CLI interface or a GUI. You can manage container networking, configurations, load balancing, service discovery, high availability, Docker host configuration and much more. An orchestrator is responsible for running, distributing, scaling and healing workloads across a collection of nodes. Typically orchestrating is provided by the same products that provide cluster infrastructure, like Kubernetes and Azure Fabric.

Python3 Lambda functions – Quick cheat sheet

On this page you will find an overview of Python 3 lambda functions, how they are used and common applications.
Python3 Lambda functions are invaluable for quick and easy data cleaning and many data related tasks and for streaming data processing.

Simple Lambda

Map Lambda

square = lambda x : x*x
print(map(square, [1,2,3,4]) #squares all elements in list

Map Lambda two lists

Filter Lambda

Define a lambda expression, which must evaluate to true, for a list element to be saved in the output list

Reduce Lambda

The reduce function allows us, to accumulate a variable, over a list of inputs. Let’s say we to implement the product function

Lambda If Else with Reduce

With this code, we can implement fizz buz in one line, but we split it up in a view lines so it is easier to understand

Azure Security – Security methods overview

There are many ways to make your cloud system more secure, here is a little overview of the most common and useful techniques to achieve safe cloud infrastrucutre

Account Shared Access Signature

The account -SAS is a Signature, that enables the client to access resources in one or more of the storage services. Everything you can do with service SAS you can do with account SAS as well. So basically the account SAS is used for delegating access to a group of services

Service Shared Access Signature

The Service SAS is a Signature which is used to delegate access to exactly one resource.

Stored Access Policy

A stored acess policy gives you more fine tunes control over service SAS on the server side. The stored acess policy (SAP) can be used to group shared access signatures and to provide additional restrictions for signatures that are bound by that policy. You can use SAP on Blob containesr, File Shares, Qoues, and Tables.

Role Based Access controll (RBAC)

RBAC lets you distribute resource access much more fine-grained than with the other methods.

Things I wish I knew, before working with Azure- Everything you should know about, before starting with Microsoft Azure!

  • What are resource groups in Azure?
    • What is a resource?
      • Any manageable item that you can rent through Azure is considered a resource. For example, virtual machines, storage accounts, web apps, databases functions and more, basically anything you create and manage in Azure
    • What is a resource provider?
      • Resource providers are the services, that supplies Azure with resources on demand. For example, Microsoft.Compute provides virtual machines. Microsoft.Storage is providing storage as the name implies. The provider gives access to operations on the resources he is providing.
    • What is a resource manager template?
      • The resource manager template defines which resources to deploy to a resource group. With templates, you can define how resources will be made available consistently and also how and which resources to release, when the system is in a critical predefined state.
    • What is a resource group?
      • Resource Groups describe a collection of all building blocks you have defined for your app. If you want to share data between apps or functions, it makes often sense to put them in the same groups, as it also makes exchanging data between them easier.
    • What does deploying a web app mean in azure context?
      • When we deploy a web app in Azure, all we do is just tell Microsoft to rent out a few computer parts for us to run our server! We can define our web app locally and then just upload it to the cloud servers, which will serve our content worldwide!
  • What are the Azure functions
    • Serverless functions in Azure can be defined very simply and connected to any app with minimal effort! The code for the function is stored on azures servers and only invoked when it is triggered by one of the many trigger mechanism. They consist of a trigger, input bindings and output binding which we will explain in detail later on
  • What are Azure Logic apps
    • Logic apps enable you to automate and orchestrate tasks. They are one of the main tools to automate processes and save you precious time! Logic apps even let you combine and concatenate multiple different apps into one! Connect everything with everyone is the motto of this set of features.
  • What is a storage account and why do I need one in Azure?
    • A storage account is a reference to all the data object stored for your account like blobs, files, ques, tables, disks and so on.
  • Redis Cache
    • Instead of renting the normal data storage or the distributed Hadoop storage, you can also rent super fast Redis Cache, which is basically just RAM and highly cachable data storage. Depending on your use case, this can be very valuable for time and efficiency critical operations
  • Power Shell / Bash Shell
    • Microsoft provides a great CLI interface to manage your cloud infrastructure
  • What are the containers?
    • A container is basically a virtualized software. Instead of having to care about the hardware and the operating system, you just ask for a container and in that one, your software project will run. The great thing about containers is, that they are hardware and OS independent, so you can just share your app container with someone and they can run your app with any issues, saving a huge amount of time when deploying software! Using such a container-based design yields more efficient architectures. Containers also let your team work faster, deploy more efficiently and operate at a much larger scale. Using a container also mean, you do not have to set up a whole VM, it is just everything you need to contain the app! This means containers are much more lightweight than VM’s This basically means, your software is decoupled from the hardware and OS, which leaves many developers with much less headache! It also makes for a clean split between infrastructure management and software logic management
  • What are Azure function triggers?
    • Since functions in Azure are serverless, we need to define a trigger, which tells Azure when to call our function. There are many possible triggers we could use, the most common ones get triggered by any changes to the Cosmos DB, the blob storage, the queue storage, and the timer.
  • What are Azure function bindings?
    • Azure function bindings basically define the input and output arguments of any function in Azure.
  • What does serverless mean? Serverless function?
    • In the context of Azure, the are serverless functions and serverless logic apps. But they still run on a server, so how are they related to serverless? The real meaning behind serverless is, that developers do not worry about the servers, it all happens automagically in the backend implemented by Microsofts engineers
  • BONUS : What is the difference between a VM and a container?
    • You can imagien the VM as virtualizing the hardware and a container is virtualizing the software

Things I wish I knew about Azure functions , before working with Azure

Every function in Azure consists of a trigger, input and output bindings and the code defining the function of course!

What are Triggers?

Triggers are mechanisms that trigger the execution of your function. You can setup triggers for a HTTP request, a database update or almost anything.

What are bindings?

The bindings define which resources our function will have access to. It will be provided as a parameter to the function

How to configure bindings and triggers?

Every function is accompanied with a function.json, whcih defines the bindings, the directions and triggers. For compiled languages, so any non scripting language, we do not have to create the function.json file ourselves, since it can be automatically generated from the function code. But for scirpting languages, we must define the function.json ourselves.

What are Durable Functions?

Durable Functions extends Azures classical functions with functions that can have a state AND are still in a serverless enviroment! Durable Functions are also nessecary, if you want to create an Orchestrator Function. The Durable Functions made up of different classical Azure Functions.

What are some Durable Function patterns?

Often, one common pattern for the Durable function is that you chain together a bunch of normal functions and their output it piped to the next function, this together. There are also Fan-out/fan-in patterns, which runs a bunch of functions in parallel and waits for all of then to finish, to return the final result. Then there is Async HTTP Api calls and liek the name implies, it enables us to make API calls that are not synchronous. Also, there is one pattern to program a human in the loob, called human interaction . You can check for more patterns the offical docs here