Published:
A few days ago I conducted a little experiment on Spark’s RDD operations. One of them was foreach operation (included as an action). Simply, this operation is applied to each rows in the RDD and the kind of operation applied is specified via a certain function. Here’s a simple example:
Published:
The problem is simple. How to find the best threshold from an ROC and PR curve that maximise a certain binary classification metric?
Published:
Just a little note about MySQL and Adminer.
Published:
Primary purpose:
Published:
Let’s take a look at the problem statement.
Published:
According to the code base, the driver status tracking feature is only implemented for standalone cluster manager. However, based on this reference, we could also poll the driver status for mesos and kubernetes (cluster deploy mode). Additionally, such a feature is also possible for YARN.
Published:
I was thinking of the following case.
Published:
A few days ago I did a small experiment with Airflow. To be precise, scheduling Airflow to run a Spark job via spark-submit to a standalone cluster. I have actually mentioned briefly about how to create a DAG and Operators in the previous post.
Published:
Airflow is basically a workflow management system. When we’re talking about “workflow”, we’re referring to a sequence of tasks that needs to be performed to accomplish a certain goal. A simple example would be related to an ordinary ETL job, such as fetching data from data sources, transforming the data into certain formats which in accordance with the requirements, and then storing the transformed data to a data warehouse.
Published:
Here we’re gonna look at how to solve the following algebra problem.
Published:
An interesting paper: http://www.phontron.com/paper/oda15ase.pdf
Published:
Let’s play with the 2nd problem of the International Mathematics Olympiad (IMO) 2012.
Published:
In the previous post, I mentioned that there are several observed points regarding Griffin during my exploration.
Published:
Primary purpose:
Published:
Primary purpose:
Published:
Primary purpose:
Published:
Primary purpose:
Published:
To discuss about this stack frame, we’ll see from Assembly language point of view.
Published:
Buffer Overflow is one of code’s exploitation technique which uses buffer weakness. In addition, buffer is a block or space for saving datas.
Published:
Woe & information value (IV) are used as a framework for attribute relevance analysis. WoE and IV can be utilised independently since each of them play different roles.
Published:
Kerberos is simply a “ticket-based” authentication protocol. It enhances the security approach used by password-based authentication protocol. Since there might be a possibility for tappers to take over the password, Kerberos mitigates this by leveraging a ticket (how it is generated is explained below) that ideally should only be known by the client and the service.
Published:
Have you ever heard of imblearn package? Based on its name, I think people who are familiar with machine learning are going to presume that it’s a package specifically created for tackling the problem of imbalanced data. If you do a deeper search, you’re gonna find its GitHub repository here. And yes, once again, it’s a Python package for playing with imbalanced data.
Published:
I came across a research paper related to balanced random forest for imbalanced data. For the sake of clarity, the following is the algorithm of BRF taken from the paper:
Published:
Livy offers a REST interface that is used to interact with Spark cluster. It provides two general approaches for job submission and monitoring.
Published:
A few days back I tried to submit a Spark job to a Livy server deployed via local mode. The procedure was straightforward since the only thing to do was to specify the job file along with the configuration parameters (like what we do when using spark-submit directly).
Published:
The Monty Hall Problem can be stated as the following:
Published:
One of the techniques in hyperparameter tuning is called Bayesian Optimization. It selects the next hyperparameter to evaluate based on the previous trials.
Published:
In the previous post, I wrote about how to perform pandas groupBy operation on a large dataset in streaming way. The main problem being addressed is optimum memory consumption since the data size might be extremely large.
Published:
I came across an article about how to perform groupBy operation for large dataset. Long story short, the author proposes an approach called streaming groupBy where the dataset is divided into chunks and the groupBy operation is applied to each chunk. This approach is implemented with pandas.
Published:
H2O provides a platform for building machine learning models in a scalable way. By focusing on scalability, it leverages the concept of cluster computing and therefore enables engineers to perform big data analytics.
Published:
This article is about how to install Cassandra and play with several of its query languages. To accomplish that, I’m going to utilize Docker.
Published:
The concept of window function in Spark is pretty interesting. One of its primary usage is calculating cumulative values. Here’s a simple example.
Published:
In BigQuery, an external data source is a data source that we can query directly although the data is not stored in BigQuery’s storage. We can query the data source just by creating an external table that refers to the data source instead of loading it to BigQuery.
Published:
Relevant Paper: Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning
Published:
Primary purpose:
Published:
Primary purpose:
Published:
Primary purpose:
Published:
Primary purpose:
Published:
Published:
In the earlier section we have learnt a bit about buffer overflow technique. The primary concept is flooding the stack frame with input exceeding the buffer limit so that we can manipulate any datas saved on the stack frame. Some things that can be done using this technique are change the return address so that the attackers can call any functions they want, change the content of variables so that the function executes corresponding code, or change the return value of a function.
Published:
To discuss about this stack frame, we’ll see from Assembly language point of view.
Published:
Buffer Overflow is one of code’s exploitation technique which uses buffer weakness. In addition, buffer is a block or space for saving datas.
Published:
In the previous post I mentioned about how to demonstrate the Wallis product for pi by starting from the powered sine integration.
Published:
The Wallis’ infinite product for pi states the following.
Published:
“Consistency, Availability, and Partition Tolerance” - choose two.
Published:
This article is about how to install Cassandra and play with several of its query languages. To accomplish that, I’m going to utilize Docker.
Published:
Applying central moment functions in Spark might be tricky, especially for skewness and kurtosis.
Published:
Let’s use a simple example data to demonstrate how XGBoost algorithm works on a classification problem.
Published:
In the previous post I mentioned about how Gradient Boosting algorithm works for a regression problem.
Published:
I was thinking of the following case.
Published:
According to the code base, the driver status tracking feature is only implemented for standalone cluster manager. However, based on this reference, we could also poll the driver status for mesos and kubernetes (cluster deploy mode). Additionally, such a feature is also possible for YARN.
Published:
Published:
I came across an odd use case when applying F.col() on certain dataframe operations on PySpark v.2.4.0. Please note that the context of this issue is on Oct 6, 2019. Such an issue might have been solved in the future.
Published:
Published:
Primary purpose:
Published:
Primary purpose:
Published:
Primary purpose:
Published:
Primary purpose:
Published:
Published:
In the earlier section we have learnt a bit about buffer overflow technique. The primary concept is flooding the stack frame with input exceeding the buffer limit so that we can manipulate any datas saved on the stack frame. Some things that can be done using this technique are change the return address so that the attackers can call any functions they want, change the content of variables so that the function executes corresponding code, or change the return value of a function.
Published:
To discuss about this stack frame, we’ll see from Assembly language point of view.
Published:
Buffer Overflow is one of code’s exploitation technique which uses buffer weakness. In addition, buffer is a block or space for saving datas.
Published:
The Monty Hall Problem can be stated as the following:
Published:
Just wanted to confirm whether the Kafka consumers were aware of new topic’s partitions.
Published:
The final problem of the International Mathematics Olympiad (IMO) 1988 is considered to be the most difficult problem on the contest.
Published:
In the previous post, we discuss about the implementation of Kalman filter for static state (the true value of the object’s states are constant over time). In addition, the Kalman filter algorithm is applied to estimate single true value.
Published:
Kalman filter is an iterative mathematical process applied on consecutive data inputs to quickly estimate the true value (position, velocity, weight, temperature, etc) of the object being measured, when the measured values contain random error or uncertainty.
Published:
In the previous post I mentioned about a simple way of estimating the density ratio of two probability distributions. I decided to create a python package that provides such a functionality.
Published:
In the previous post I shared about how to detect covariate shift with a simple technique–model based approach. After knowing that the data distribution changes, what can we do to address such an issue?
Published:
Covariate shift happens when the distribution of train data differs with the distribution of test data. Take a look at the following probability equation.
Published:
I encountered an issue when applying crosstab function in PySpark to a pretty big data. And I think this should be considered as a pretty big issue. Please note that the context of this issue is on Sep 20, 2019. Such an issue might have been solved in the future.
Published:
Kolmogorov-Smirnov (KS) test is a non-parametric test for the equality of probability distributions.
Published:
I used kafka-python v.1.4.7 as the client.
Published:
Code profiling is simply used to assess the code performance, including its functions and sub-functions within functions. One of its obvious usage is code optimisation where a developer wants to improve the code efficiency by searching for the bottlenecks in the code.
Published:
Kolmogorov-Smirnov (KS) test is a non-parametric test for the equality of probability distributions.
Published:
“If you torture the data long enough, it will confess to anything” - Ronald Coase.
Published:
In the previous post, I mentioned that there are several observed points regarding Griffin during my exploration.
Published:
A few days back I was exploring a big data quality tool called Griffin. There are lots of DQ tools out there, such as Deequ, Target’s data validator, Tensorflow data validator, PySpark Owl, and Great Expectation. There’s another one called Cerberus. It doesn’t natively support large-scale data however.
Published:
In the previous post, I mentioned about the general formula of the H statistic is the following (Source: Wikipedia - Kruskal–Wallis one-way analysis of variance):
Published:
The Kruskal-Wallis test is a non-parametric statistical test that is used to evaluate whether the medians of two or more groups are different. Since the test is non-parametric, it doesn’t assume that the data comes from a particular distribution.
Published:
A few days ago I did a little exploration on Spark’s groupBy behavior. Precisely, I wanted to see whether the order of the data was still preserved when applying groupBy on a repartitioned dataframe.
Published:
In this post, we’re going to look at how to set up a database along with the tables in Hive.
Published:
Recently I was exploring ways of adding a unique row ID column to a dataframe. The requirement is simple: “the row ID should strictly increase with difference of one and the data order is not modified”.
Published:
A few days ago I came across a case where I needed to define a dataframe’s column name with a special character, that is a dot (‘.’). Take a look at thee following schema example.
Published:
A nested column is basically just a column with one or more sub-columns. Take a look at the following example.
Published:
I came across an odd use case when applying F.col() on certain dataframe operations on PySpark v.2.4.0. Please note that the context of this issue is on Oct 6, 2019. Such an issue might have been solved in the future.
Published:
I encountered an issue when applying crosstab function in PySpark to a pretty big data. And I think this should be considered as a pretty big issue. Please note that the context of this issue is on Sep 20, 2019. Such an issue might have been solved in the future.
Published:
If you read my previous article titled Union Operation After Left-anti Join Might Result in Inconsistent Attributes Data, it was shown that the attributes data was inconsistent when combining two data frames after inner-join. According to the article, the solution is really simple. We just need to reorder the attributes order by using select command. Here’s a simple example.
Published:
Have you ever encountered a case where you need to compute the sum of a certain one-item operation? Consider the following example.
Published:
Unioning two dataframes after joining them with left_anti? Well, seems like a straightforward approach. However, recently I encountered a case where join operation might shift the location of the join key in the resulting dataframe. This, unfortunately, makes the dataframe’s merging result inconsistent in terms of the data in each attribute.
Published:
I encountered an intriguing result when joining a dataframe with itself (self-join). As you might have already known, one of the problems occurred when doing a self-join relates to duplicated column names. Because of this duplication, there’s an ambiguity when we do operations requiring us to provide the column names.
Published:
An intriguing question popped into my mind. After unioning several dataframes, how many partitions the resulting dataframe will have?
Published:
Have you ever wondered how the size of a dataframe can be discovered? Perhaps it sounds not so fancy thing to know, yet I think there are certain cases requiring us to have pre-knowledge of the size of our dataframe. One of them is when we want to apply broadcast operation. As you might’ve already knownn, broadcasting requires the dataframe to be small enough to fit in memory in each executor. This implicitly means that we should know about the size of the dataframe beforehand in order for broadcasting to be applied successfully. Just FYI, broadcasting enables us to configure the maximum size of a dataframe that can be pushed into each executor. Precisely, this maximum size can be configured via spark.conf.set(“spark.sql.autoBroadcastJoinThreshold”, MAX_SIZE).
Published:
In Spark, data shuffling simply means data movement. In a single machine with multiple partitions, data shuffling means that data move from one partition to another partition. Meanwhile, in multiple machines, data shuffling can have two kinds of work. The first one is data move from one partition (A) to another partition (B) within the same machine (M1), while the second one is data move from partition B to another partition (C) within different machine (M2). Data in partition C might be moved to another partition within different machine again (M3).
Published:
The concept of window function in Spark is pretty interesting. One of its primary usage is calculating cumulative values. Here’s a simple example.
Published:
Parquet is a file format with columnar style. Columnar style means that we don’t store the content of each row of the data. Here’s a simple example.
Published:
One of the characteristics of Spark that makes me interested to explore this framework further is its lazy evaluation approach. Simply put, Spark won’t execute the transformation until an action is called. I think it’s logical since when we only specify the transformation plan and don’t ask it to execute the plan, why it needs to force itself to do the computation on the data? In addition, by implementing this lazy evaluation approach, Spark might be able to optimize the logical plan. The task of making the query to be more efficient manually might be reduced significantly. Cool, right?
Published:
A few days ago I did a little exploration on Spark’s groupBy behavior. Precisely, I wanted to see whether the order of the data was still preserved when applying groupBy on a repartitioned dataframe.
Published:
A statement I encountered a few days ago: “Avoid to use Resilient Distributed Datasets (RDDs) and use Dataframes/Datasets (DFs/DTs) instead, especially in production stage”.
Published:
I was implementing a paper related to balanced random forest (BRF). Just FYI, a BRF consists of some decision trees where each tree receives instances with a ratio of 1:1 for minority and majority class. A BRF also uses m features selected randomly to determine the best split.
Published:
Spark has two types of partitioning. The first one is coalesce, while the second one is repartition.
Published:
I made a silly mistake a few days ago - well, yes.
Published:
Recently I watched a YouTube video about the infinite hotel paradox which was introduced in 1920s by a German mathematician, David Hilbert. In case you’re curious about he video, just search on YouTube using “The Infinite Hotel Paradox” keyword.
Published:
I encountered an intriguing result when joining a dataframe with itself (self-join). As you might have already known, one of the problems occurred when doing a self-join relates to duplicated column names. Because of this duplication, there’s an ambiguity when we do operations requiring us to provide the column names.
Published:
Imagine that you have some data x1, x2, x3, ..., xn originating from an unknown continuous distribution f. You’d like to estimate f.
Published:
In the previous post I shared about how to detect covariate shift with a simple technique–model based approach. After knowing that the data distribution changes, what can we do to address such an issue?
Published:
In the previous post I mentioned about a simple way of estimating the density ratio of two probability distributions. I decided to create a python package that provides such a functionality.
Published:
The initial question that popped up in my mind was how to make LIME performs faster. This should be useful enough when the data to explain is big enough.
Published:
Spark functions (UDFs) are simply functions created to overcome speed performance problem when you want to process a dataframe. It’d be useful when your Python functions were so slow in processing a dataframe in large scale. When you use a Python function, it will process the dataframe with one-row-at-a-time manner, meaning that the process would be executed sequentially. Meanwhile, if you use a Spark UDF, Spark will distribute the dataframe and the Spark UDF to the provided executors. Hence, the dataframe processing would be executed in parallel. For more information about Spark UDF, please take a look at this post.
Published:
In the previous post, I wrote about how to make LIME run in pseudo-distributed mode with PySpark UDF.
Published:
“Consistency, Availability, and Partition Tolerance” - choose two.
Published:
If in the probability context we state that P(x1, x2, ..., xn | params) means the probability of getting a set of observations x1, x2, …, and xn given the distribution parameters, then in the likelihood context we get the following.
Published:
In the previous post, I mentioned about the basic concept of two-sample Kolmogorov-Smirnov (KS) test and its implementation in Spark (Scala API).
Published:
This article is about how to install Cassandra and play with several of its query languages. To accomplish that, I’m going to utilize Docker.
Published:
The best way to try new technologies without having clutter? Docker.
Published:
A few days ago I came across a case where I needed to define a dataframe’s column name with a special character, that is a dot (‘.’). Take a look at thee following schema example.
Published:
On November 15th, 2018, I promised myself I would write down my journey of accomplishing one of my dreams. This post is the realization of that word.
Published:
I was thinking of the following case.
Published:
There are several ways of removing duplicate rows in Spark. Two of them are by using distinct() and dropDuplicates(). The former lets us to remove rows with the same values on all the columns. Meanwhile, the latter lets us to remove rows with the same values on multiple selected columns.
Published:
There are several critical issues that present when using Spark. One of them relates to data loss when a failure occurs.
Published:
In the previous post, we discuss about the implementation of Kalman filter for static state (the true value of the object’s states are constant over time). In addition, the Kalman filter algorithm is applied to estimate single true value.
Published:
Kolmogorov-Smirnov (KS) test is a non-parametric test for the equality of probability distributions.
Published:
In this post, we’re going to look at how Gradient Boosting algorithm works in a regression problem.
Published:
To me, prime numbers are really interesting in terms of their position as the building blocks of other numbers. According to the Fundamental Theorem of Arithmetic, every positive integer N can be written as a product of P1, P2, P3, …, and Pk where Pi are all prime numbers.
Published:
Given an infinite series of inverse squares of the natural numbers, what is the sum?
Published:
In the previous post I mentioned about how to demonstrate the Wallis product for pi by starting from the powered sine integration.
Published:
Euler’s formula is stated as the following.
Published:
A few days ago I did a small experiment with Airflow. To be precise, scheduling Airflow to run a Spark job via spark-submit to a standalone cluster. I have actually mentioned briefly about how to create a DAG and Operators in the previous post.
Published:
LIME is a python library used to explain predictions from any machine learning classifier.
Published:
LIME is a python library used to explain predictions from any machine learning classifier.
Published:
In BigQuery, an external data source is a data source that we can query directly although the data is not stored in BigQuery’s storage. We can query the data source just by creating an external table that refers to the data source instead of loading it to BigQuery.
Published:
A few days ago I came across a case where a module needs access to the resources directory.
Published:
Here we’re gonna look at how to solve the following algebra problem.
Published:
Let’s use a simple example data to demonstrate how XGBoost algorithm works on a classification problem.
Published:
One of the evaluation metrics that is often optimised is ROC-AUC. In this post, we’re going to discuss how an ROC curve is created.
Published:
Published:
Have you ever encountered a case where you need to compute the sum of a certain one-item operation? Consider the following example.
Published:
In BigQuery, an external data source is a data source that we can query directly although the data is not stored in BigQuery’s storage. We can query the data source just by creating an external table that refers to the data source instead of loading it to BigQuery.
Published:
Primary purpose:
Published:
In the previous post I mentioned about how Gradient Boosting algorithm works for a regression problem.
Published:
In this post, we’re going to look at how Gradient Boosting algorithm works in a regression problem.
Published:
A few days back I was exploring a big data quality tool called Griffin. There are lots of DQ tools out there, such as Deequ, Target’s data validator, Tensorflow data validator, PySpark Owl, and Great Expectation. There’s another one called Cerberus. It doesn’t natively support large-scale data however.
Published:
There are several ways of removing duplicate rows in Spark. Two of them are by using distinct() and dropDuplicates(). The former lets us to remove rows with the same values on all the columns. Meanwhile, the latter lets us to remove rows with the same values on multiple selected columns.
Published:
I came across an article about how to perform groupBy operation for large dataset. Long story short, the author proposes an approach called streaming groupBy where the dataset is divided into chunks and the groupBy operation is applied to each chunk. This approach is implemented with pandas.
Published:
I encountered an issue when applying crosstab function in PySpark to a pretty big data. And I think this should be considered as a pretty big issue. Please note that the context of this issue is on Sep 20, 2019. Such an issue might have been solved in the future.
Published:
A few days ago I did a little exploration on Spark’s groupBy behavior. Precisely, I wanted to see whether the order of the data was still preserved when applying groupBy on a repartitioned dataframe.
Published:
H2O provides a platform for building machine learning models in a scalable way. By focusing on scalability, it leverages the concept of cluster computing and therefore enables engineers to perform big data analytics.
Published:
Basically, Metabase’s SparkSQL only allows users to access data in the Hive warehouse. In other words, the data must be in Hive table format to be able to be loaded.
Published:
In this post, we’re going to look at how to set up a database along with the tables in Hive.
Published:
I used kafka-python v.1.4.7 as the client.
Published:
Published:
Application monitoring is critically important, especially when we encounter performance issues. In Spark, one way to monitor a Spark application is via Spark UI. The problem is, this Spark UI can only be accessed when the application is running.
Published:
Basically, Metabase’s SparkSQL only allows users to access data in the Hive warehouse. In other words, the data must be in Hive table format to be able to be loaded.
Published:
In this post, we’re going to look at how to set up a database along with the tables in Hive.
Published:
Primary purpose:
Published:
One of the techniques in hyperparameter tuning is called Bayesian Optimization. It selects the next hyperparameter to evaluate based on the previous trials.
Published:
In the previous post, I mentioned about the general formula of the H statistic is the following (Source: Wikipedia - Kruskal–Wallis one-way analysis of variance):
Published:
The Kruskal-Wallis test is a non-parametric statistical test that is used to evaluate whether the medians of two or more groups are different. Since the test is non-parametric, it doesn’t assume that the data comes from a particular distribution.
Published:
It’s quite bothering when reading a publication that only provides a “statistically significant” result without telling much about the analysis prior to conducting the experiment.
Published:
“If you torture the data long enough, it will confess to anything” - Ronald Coase.
Published:
Have you ever heard of imblearn package? Based on its name, I think people who are familiar with machine learning are going to presume that it’s a package specifically created for tackling the problem of imbalanced data. If you do a deeper search, you’re gonna find its GitHub repository here. And yes, once again, it’s a Python package for playing with imbalanced data.
Published:
Let’s play with the 2nd problem of the International Mathematics Olympiad (IMO) 2012.
Published:
The final problem of the International Mathematics Olympiad (IMO) 1988 is considered to be the most difficult problem on the contest.
Published:
In the previous post I shared about how to detect covariate shift with a simple technique–model based approach. After knowing that the data distribution changes, what can we do to address such an issue?
Published:
In the previous post, I wrote about how to perform pandas groupBy operation on a large dataset in streaming way. The main problem being addressed is optimum memory consumption since the data size might be extremely large.
Published:
Let’s play with the 2nd problem of the International Mathematics Olympiad (IMO) 2012.
Published:
Recently I watched a YouTube video about the infinite hotel paradox which was introduced in 1920s by a German mathematician, David Hilbert. In case you’re curious about he video, just search on YouTube using “The Infinite Hotel Paradox” keyword.
Published:
To me, prime numbers are really interesting in terms of their position as the building blocks of other numbers. According to the Fundamental Theorem of Arithmetic, every positive integer N can be written as a product of P1, P2, P3, …, and Pk where Pi are all prime numbers.
Published:
Last time I wrote about the infinite products representation for pi that is regarded as the Wallis’ product for pi.
Published:
Recently I watched a YouTube video about the infinite hotel paradox which was introduced in 1920s by a German mathematician, David Hilbert. In case you’re curious about he video, just search on YouTube using “The Infinite Hotel Paradox” keyword.
Published:
Woe & information value (IV) are used as a framework for attribute relevance analysis. WoE and IV can be utilised independently since each of them play different roles.
Published:
Recently I played with a simple Spark Streaming application. Precisely, I investigated the behavior of repartitioning on different level of input data streams. For instance, we have two input data streams, such as linesDStream and wordsDStream. The question is, is the repartitioning result different if I repartition after linesDStream and after wordsDStream?
Published:
LIME is a python library used to explain predictions from any machine learning classifier.
Published:
Published:
Published:
I used kafka-python v.1.4.7 as the client.
Published:
In the previous article about Kafka Consumer Awareness of New Topic Partitions, I wrote about partitions balancing by Kafka consumers. In other words, I’d like to see whether Kafka consumers are aware of new topic partitions.
Published:
Just wanted to confirm whether the Kafka consumers were aware of new topic’s partitions.
Published:
Basically, you can presume Kafka as a messaging system. When an application sends a message to another application, one thing they need to do is to specify how to send the message. The most obvious use case in using a messaging system, in my opinion, is when we’re dealing with big data. For instance, a sender application shares a large amount of data that need to be processed by a receiver application. However, the processing rate by the receiver is lower than the sending rate. Consequently, the receiver might be overloaded since it’s unable to receive messages anymore while the processing is running. Although we’re using distributed receivers, we still have to tell the sender about which receiver node it should send the message to.
Published:
In the previous post, we discuss about the implementation of Kalman filter for static state (the true value of the object’s states are constant over time). In addition, the Kalman filter algorithm is applied to estimate single true value.
Published:
Kalman filter is an iterative mathematical process applied on consecutive data inputs to quickly estimate the true value (position, velocity, weight, temperature, etc) of the object being measured, when the measured values contain random error or uncertainty.
Published:
Kerberos is simply a “ticket-based” authentication protocol. It enhances the security approach used by password-based authentication protocol. Since there might be a possibility for tappers to take over the password, Kerberos mitigates this by leveraging a ticket (how it is generated is explained below) that ideally should only be known by the client and the service.
Published:
In the previous post, I mentioned about the basic concept of two-sample Kolmogorov-Smirnov (KS) test and its implementation in Spark (Scala API).
Published:
Kolmogorov-Smirnov (KS) test is a non-parametric test for the equality of probability distributions.
Published:
In the previous post, I mentioned about the general formula of the H statistic is the following (Source: Wikipedia - Kruskal–Wallis one-way analysis of variance):
Published:
The Kruskal-Wallis test is a non-parametric statistical test that is used to evaluate whether the medians of two or more groups are different. Since the test is non-parametric, it doesn’t assume that the data comes from a particular distribution.
Published:
In the previous post, I mentioned about the basic concept of two-sample Kolmogorov-Smirnov (KS) test and its implementation in Spark (Scala API).
Published:
Applying central moment functions in Spark might be tricky, especially for skewness and kurtosis.
Published:
Published:
One of the characteristics of Spark that makes me interested to explore this framework further is its lazy evaluation approach. Simply put, Spark won’t execute the transformation until an action is called. I think it’s logical since when we only specify the transformation plan and don’t ask it to execute the plan, why it needs to force itself to do the computation on the data? In addition, by implementing this lazy evaluation approach, Spark might be able to optimize the logical plan. The task of making the query to be more efficient manually might be reduced significantly. Cool, right?
Published:
Unioning two dataframes after joining them with left_anti? Well, seems like a straightforward approach. However, recently I encountered a case where join operation might shift the location of the join key in the resulting dataframe. This, unfortunately, makes the dataframe’s merging result inconsistent in terms of the data in each attribute.
Published:
The final problem of the International Mathematics Olympiad (IMO) 1988 is considered to be the most difficult problem on the contest.
Published:
In the previous post I wrote about how to derive the Levinson-Durbin recursion.
Published:
I was exploring about the Levinson-Durbin recursion and curious about the formula’s derivation.
Published:
On November 15th, 2018, I promised myself I would write down my journey of accomplishing one of my dreams. This post is the realization of that word.
Published:
LIME is a python library used to explain predictions from any machine learning classifier.
Published:
In the previous post, I wrote about how to make LIME run in pseudo-distributed mode with PySpark UDF.
Published:
The initial question that popped up in my mind was how to make LIME performs faster. This should be useful enough when the data to explain is big enough.
Published:
Livy offers a REST interface that is used to interact with Spark cluster. It provides two general approaches for job submission and monitoring.
Published:
A few days back I tried to submit a Spark job to a Livy server deployed via local mode. The procedure was straightforward since the only thing to do was to specify the job file along with the configuration parameters (like what we do when using spark-submit directly).
Published:
LIME is a python library used to explain predictions from any machine learning classifier.
Published:
Below is a script for running spark via spark-submit (local mode) that utilizes logging.
Published:
Below is a script for running spark via spark-submit (local mode) that utilizes logging.
Published:
This article is about a brief overview of how to store several of the most recent log files in PySpark logging.
Published:
This article is about a brief overview of how to write log messages using PySpark logging.
Published:
Let’s use a simple example data to demonstrate how XGBoost algorithm works on a classification problem.
Published:
In the previous post I mentioned about how Gradient Boosting algorithm works for a regression problem.
Published:
LIME is a python library used to explain predictions from any machine learning classifier.
Published:
Imagine that you have some data x1, x2, x3, ..., xn originating from an unknown continuous distribution f. You’d like to estimate f.
Published:
One of the techniques in hyperparameter tuning is called Bayesian Optimization. It selects the next hyperparameter to evaluate based on the previous trials.
Published:
In this post, we’re going to look at how Gradient Boosting algorithm works in a regression problem.
Published:
In the previous post, I mentioned about the basic concept of maximum likelihood estimation (MLE). Please visit the post if you need a refresher.
Published:
In the previous post, I mentioned about the basic concept of two-sample Kolmogorov-Smirnov (KS) test and its implementation in Spark (Scala API).
Published:
Kolmogorov-Smirnov (KS) test is a non-parametric test for the equality of probability distributions.
Published:
In the previous post, I mentioned that there are several observed points regarding Griffin during my exploration.
Published:
A few days back I was exploring a big data quality tool called Griffin. There are lots of DQ tools out there, such as Deequ, Target’s data validator, Tensorflow data validator, PySpark Owl, and Great Expectation. There’s another one called Cerberus. It doesn’t natively support large-scale data however.
Published:
Suppose we conduct K experiments on a kind of measurement. On each experiment, we take N observations. In other words, we’ll have N * K data at the end.
Published:
I was experimenting with the weight of evidence (WoE) encoding for continuous data. The preparation is quite different from categorical data in terms of binning characteristics.
Published:
In the previous post, I mentioned about how collinearity affects the computation of the beta estimators.
Published:
In simple terms, we could define collinearity as a condition where two variables are highly correlated (positively / negatively). When there are more than two variables, it’s sometimes referred as multicollinearity.
Published:
Woe & information value (IV) are used as a framework for attribute relevance analysis. WoE and IV can be utilised independently since each of them play different roles.
Published:
In the previous post I mentioned about a simple way of estimating the density ratio of two probability distributions. I decided to create a python package that provides such a functionality.
Published:
In the previous post I shared about how to detect covariate shift with a simple technique–model based approach. After knowing that the data distribution changes, what can we do to address such an issue?
Published:
Covariate shift happens when the distribution of train data differs with the distribution of test data. Take a look at the following probability equation.
Published:
In the previous post, I wrote about how to make LIME run in pseudo-distributed mode with PySpark UDF.
Published:
One of the evaluation metrics that is often optimised is ROC-AUC. In this post, we’re going to discuss how an ROC curve is created.
Published:
The problem is simple. How to find the best threshold from an ROC and PR curve that maximise a certain binary classification metric?
Published:
H2O provides a platform for building machine learning models in a scalable way. By focusing on scalability, it leverages the concept of cluster computing and therefore enables engineers to perform big data analytics.
Published:
Have you ever heard of imblearn package? Based on its name, I think people who are familiar with machine learning are going to presume that it’s a package specifically created for tackling the problem of imbalanced data. If you do a deeper search, you’re gonna find its GitHub repository here. And yes, once again, it’s a Python package for playing with imbalanced data.
Published:
I was implementing a paper related to balanced random forest (BRF). Just FYI, a BRF consists of some decision trees where each tree receives instances with a ratio of 1:1 for minority and majority class. A BRF also uses m features selected randomly to determine the best split.
Published:
I came across a research paper related to balanced random forest for imbalanced data. For the sake of clarity, the following is the algorithm of BRF taken from the paper:
Published:
Relevant Paper: Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning
Published:
First article in 2019.
Published:
Published:
Published:
Published:
Published:
Published:
Euler’s formula is stated as the following.
Published:
There might be a case where we need to perform a certain operation on each data partition. One of the most common examples is the use of mapPartitions. Sometimes, such an operation probably requires a more complicated procedure. This, in the end, makes the method executing the operation needs more than one parameter.
Published:
Here we’re gonna look at how to solve the following algebra problem.
Published:
Let’s take a look at the problem statement.
Published:
The final problem of the International Mathematics Olympiad (IMO) 1988 is considered to be the most difficult problem on the contest.
Published:
Euler’s formula is stated as the following.
Published:
Given an infinite series of inverse squares of the natural numbers, what is the sum?
Published:
Last time I wrote about the infinite products representation for pi that is regarded as the Wallis’ product for pi.
Published:
In the previous post I mentioned about how to demonstrate the Wallis product for pi by starting from the powered sine integration.
Published:
The Wallis’ infinite product for pi states the following.
Published:
Recently I watched a YouTube video about the infinite hotel paradox which was introduced in 1920s by a German mathematician, David Hilbert. In case you’re curious about he video, just search on YouTube using “The Infinite Hotel Paradox” keyword.
Published:
Let’s play with the 2nd problem of the International Mathematics Olympiad (IMO) 2012.
Published:
In the previous post I wrote about how to derive the Levinson-Durbin recursion.
Published:
I was exploring about the Levinson-Durbin recursion and curious about the formula’s derivation.
Published:
As the name suggests, moment generating function (MGF) provides a function that generates moments, such as E[X], E[X^2], E[X^3], and so forth.
Published:
Applying central moment functions in Spark might be tricky, especially for skewness and kurtosis.
Published:
One sample z-test is used to examine whether the difference between a population mean and a certain value is significant.
Published:
The theorem of nested cube roots (Ramanujan) states the following.
Published:
In the previous post, I mentioned about how collinearity affects the computation of the beta estimators.
Published:
In simple terms, we could define collinearity as a condition where two variables are highly correlated (positively / negatively). When there are more than two variables, it’s sometimes referred as multicollinearity.
Published:
Yesterday I came across an interesting Math paper discussing about the Riemann hypothesis. Regarding the concept itself, there’s lots of maths but I think I enjoyed the reading. Frankly speaking, although mathematics is one of my favourite subjects, I’ve been rarely playing with it (esp. pure maths) since I got acquainted with AI and big data engineering world. Now I think it’s just fine to play with it again. Just for fun.
Published:
In the previous post, I mentioned about the basic concept of maximum likelihood estimation (MLE). Please visit the post if you need a refresher.
Published:
If in the probability context we state that P(x1, x2, ..., xn | params) means the probability of getting a set of observations x1, x2, …, and xn given the distribution parameters, then in the likelihood context we get the following.
Published:
One sample z-test is used to examine whether the difference between a population mean and a certain value is significant.
Published:
Basically, you can presume Kafka as a messaging system. When an application sends a message to another application, one thing they need to do is to specify how to send the message. The most obvious use case in using a messaging system, in my opinion, is when we’re dealing with big data. For instance, a sender application shares a large amount of data that need to be processed by a receiver application. However, the processing rate by the receiver is lower than the sending rate. Consequently, the receiver might be overloaded since it’s unable to receive messages anymore while the processing is running. Although we’re using distributed receivers, we still have to tell the sender about which receiver node it should send the message to.
Published:
Basically, Metabase’s SparkSQL only allows users to access data in the Hive warehouse. In other words, the data must be in Hive table format to be able to be loaded.
Published:
I used kafka-python v.1.4.7 as the client.
Published:
In the previous article about Kafka Consumer Awareness of New Topic Partitions, I wrote about partitions balancing by Kafka consumers. In other words, I’d like to see whether Kafka consumers are aware of new topic partitions.
Published:
If in the probability context we state that P(x1, x2, ..., xn | params) means the probability of getting a set of observations x1, x2, …, and xn given the distribution parameters, then in the likelihood context we get the following.
Published:
In the previous post, I wrote about how to make LIME run in pseudo-distributed mode with PySpark UDF.
Published:
LIME is a python library used to explain predictions from any machine learning classifier.
Published:
As the name suggests, moment generating function (MGF) provides a function that generates moments, such as E[X], E[X^2], E[X^3], and so forth.
Published:
As the name suggests, moment generating function (MGF) provides a function that generates moments, such as E[X], E[X^2], E[X^3], and so forth.
Published:
Here’s the scenario.
Published:
Application monitoring is critically important, especially when we encounter performance issues. In Spark, one way to monitor a Spark application is via Spark UI. The problem is, this Spark UI can only be accessed when the application is running.
Published:
I was experimenting with the weight of evidence (WoE) encoding for continuous data. The preparation is quite different from categorical data in terms of binning characteristics.
Published:
The Monty Hall Problem can be stated as the following:
Published:
In the previous post, I mentioned about how collinearity affects the computation of the beta estimators.
Published:
In simple terms, we could define collinearity as a condition where two variables are highly correlated (positively / negatively). When there are more than two variables, it’s sometimes referred as multicollinearity.
Published:
In the previous post, we discuss about the implementation of Kalman filter for static state (the true value of the object’s states are constant over time). In addition, the Kalman filter algorithm is applied to estimate single true value.
Published:
Whenever we call dataframe.writeStream.start() in structured streaming, Spark creates a new stream that reads from a data source (specified by dataframe.readStream). The data passed through the stream is then processed (if needed) and sinked to a certain location.
Published:
Just a little note about MySQL and Adminer.
Published:
Basically, Metabase’s SparkSQL only allows users to access data in the Hive warehouse. In other words, the data must be in Hive table format to be able to be loaded.
Published:
An interesting paper: http://www.phontron.com/paper/oda15ase.pdf
Published:
Published:
Published:
Published:
A nested column is basically just a column with one or more sub-columns. Take a look at the following example.
Published:
The theorem of nested cube roots (Ramanujan) states the following.
Published:
In the previous post, I mentioned about the general formula of the H statistic is the following (Source: Wikipedia - Kruskal–Wallis one-way analysis of variance):
Published:
The Kruskal-Wallis test is a non-parametric statistical test that is used to evaluate whether the medians of two or more groups are different. Since the test is non-parametric, it doesn’t assume that the data comes from a particular distribution.
Published:
In the previous post, I mentioned about the basic concept of maximum likelihood estimation (MLE). Please visit the post if you need a refresher.
Published:
This article is about how to install Cassandra and play with several of its query languages. To accomplish that, I’m going to utilize Docker.
Published:
The final problem of the International Mathematics Olympiad (IMO) 1988 is considered to be the most difficult problem on the contest.
Published:
I think one of the unique features provided by PyArmor is that it lets the users to configure the ways to obfuscate the codes.
Published:
Basically, code obfuscation is a technique used to modify the source code so that it becomes difficult to understand but remains fully functional. The main objective is to protect intellectual properties and prevent hackers from reverse engineering a proprietary source code.
Published:
The final problem of the International Mathematics Olympiad (IMO) 1988 is considered to be the most difficult problem on the contest.
Published:
Relevant Paper: Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning
Published:
First article in 2019.
Published:
“If you torture the data long enough, it will confess to anything” - Ronald Coase.
Published:
One sample z-test is used to examine whether the difference between a population mean and a certain value is significant.
Published:
I came across an article about how to perform groupBy operation for large dataset. Long story short, the author proposes an approach called streaming groupBy where the dataset is divided into chunks and the groupBy operation is applied to each chunk. This approach is implemented with pandas.
Published:
Spark functions (UDFs) are simply functions created to overcome speed performance problem when you want to process a dataframe. It’d be useful when your Python functions were so slow in processing a dataframe in large scale. When you use a Python function, it will process the dataframe with one-row-at-a-time manner, meaning that the process would be executed sequentially. Meanwhile, if you use a Spark UDF, Spark will distribute the dataframe and the Spark UDF to the provided executors. Hence, the dataframe processing would be executed in parallel. For more information about Spark UDF, please take a look at this post.
Published:
Recently I watched a YouTube video about the infinite hotel paradox which was introduced in 1920s by a German mathematician, David Hilbert. In case you’re curious about he video, just search on YouTube using “The Infinite Hotel Paradox” keyword.
Published:
Whenever we call dataframe.writeStream.start() in structured streaming, Spark creates a new stream that reads from a data source (specified by dataframe.readStream). The data passed through the stream is then processed (if needed) and sinked to a certain location.
Published:
I was curious about how checkpoint files in Spark structured streaming looked like. To introduce the basic concept, checkpointing simply denotes the progress information of streaming process. This checkpoint files are usually used for failure recovery. More detail explanation can be found here.
Published:
There are several critical issues that present when using Spark. One of them relates to data loss when a failure occurs.
Published:
Parquet is a file format with columnar style. Columnar style means that we don’t store the content of each row of the data. Here’s a simple example.
Published:
I used kafka-python v.1.4.7 as the client.
Published:
In the previous article about Kafka Consumer Awareness of New Topic Partitions, I wrote about partitions balancing by Kafka consumers. In other words, I’d like to see whether Kafka consumers are aware of new topic partitions.
Published:
Just wanted to confirm whether the Kafka consumers were aware of new topic’s partitions.
Published:
Parquet is a file format with columnar style. Columnar style means that we don’t store the content of each row of the data. Here’s a simple example.
Published:
In Spark, data shuffling simply means data movement. In a single machine with multiple partitions, data shuffling means that data move from one partition to another partition. Meanwhile, in multiple machines, data shuffling can have two kinds of work. The first one is data move from one partition (A) to another partition (B) within the same machine (M1), while the second one is data move from partition B to another partition (C) within different machine (M2). Data in partition C might be moved to another partition within different machine again (M3).
Published:
The problem is really simple. After equi-joining (inner) two dataframes, a certain operation is applied to each partition. Precisely, such an operation can be accomplished by the following code:
Published:
Imagine that you have some data x1, x2, x3, ..., xn originating from an unknown continuous distribution f. You’d like to estimate f.
Published:
Here’s the scenario.
Published:
Application monitoring is critically important, especially when we encounter performance issues. In Spark, one way to monitor a Spark application is via Spark UI. The problem is, this Spark UI can only be accessed when the application is running.
Published:
Given an infinite series of inverse squares of the natural numbers, what is the sum?
Published:
Last time I wrote about the infinite products representation for pi that is regarded as the Wallis’ product for pi.
Published:
I encountered an issue when applying crosstab function in PySpark to a pretty big data. And I think this should be considered as a pretty big issue. Please note that the context of this issue is on Sep 20, 2019. Such an issue might have been solved in the future.
Published:
The problem is simple. How to find the best threshold from an ROC and PR curve that maximise a certain binary classification metric?
Published:
Yesterday I came across an interesting Math paper discussing about the Riemann hypothesis. Regarding the concept itself, there’s lots of maths but I think I enjoyed the reading. Frankly speaking, although mathematics is one of my favourite subjects, I’ve been rarely playing with it (esp. pure maths) since I got acquainted with AI and big data engineering world. Now I think it’s just fine to play with it again. Just for fun.
Published:
In the previous post I mentioned about a simple way of estimating the density ratio of two probability distributions. I decided to create a python package that provides such a functionality.
Published:
In the previous post I shared about how to detect covariate shift with a simple technique–model based approach. After knowing that the data distribution changes, what can we do to address such an issue?
Published:
If in the probability context we state that P(x1, x2, ..., xn | params) means the probability of getting a set of observations x1, x2, …, and xn given the distribution parameters, then in the likelihood context we get the following.
Published:
The Monty Hall Problem can be stated as the following:
Published:
In the previous article about Kafka Consumer Awareness of New Topic Partitions, I wrote about partitions balancing by Kafka consumers. In other words, I’d like to see whether Kafka consumers are aware of new topic partitions.
Published:
Just wanted to confirm whether the Kafka consumers were aware of new topic’s partitions.
Published:
Code profiling is simply used to assess the code performance, including its functions and sub-functions within functions. One of its obvious usage is code optimisation where a developer wants to improve the code efficiency by searching for the bottlenecks in the code.
Published:
I think one of the unique features provided by PyArmor is that it lets the users to configure the ways to obfuscate the codes.
Published:
Basically, code obfuscation is a technique used to modify the source code so that it becomes difficult to understand but remains fully functional. The main objective is to protect intellectual properties and prevent hackers from reverse engineering a proprietary source code.
Published:
Recently I was exploring ways of adding a unique row ID column to a dataframe. The requirement is simple: “the row ID should strictly increase with difference of one and the data order is not modified”.
Published:
I came across an odd use case when applying F.col() on certain dataframe operations on PySpark v.2.4.0. Please note that the context of this issue is on Oct 6, 2019. Such an issue might have been solved in the future.
Published:
I encountered an issue when applying crosstab function in PySpark to a pretty big data. And I think this should be considered as a pretty big issue. Please note that the context of this issue is on Sep 20, 2019. Such an issue might have been solved in the future.
Published:
There are several critical issues that present when using Spark. One of them relates to data loss when a failure occurs.
Published:
If you read my previous article titled Union Operation After Left-anti Join Might Result in Inconsistent Attributes Data, it was shown that the attributes data was inconsistent when combining two data frames after inner-join. According to the article, the solution is really simple. We just need to reorder the attributes order by using select command. Here’s a simple example.
Published:
This article is about a brief overview of how to write log messages using PySpark logging.
Published:
I think one of the unique features provided by PyArmor is that it lets the users to configure the ways to obfuscate the codes.
Published:
Basically, code obfuscation is a technique used to modify the source code so that it becomes difficult to understand but remains fully functional. The main objective is to protect intellectual properties and prevent hackers from reverse engineering a proprietary source code.
Published:
Airflow is basically a workflow management system. When we’re talking about “workflow”, we’re referring to a sequence of tasks that needs to be performed to accomplish a certain goal. A simple example would be related to an ordinary ETL job, such as fetching data from data sources, transforming the data into certain formats which in accordance with the requirements, and then storing the transformed data to a data warehouse.
Published:
Just wanted to confirm whether the Kafka consumers were aware of new topic’s partitions.
Published:
There might be a case where we need to perform a certain operation on each data partition. One of the most common examples is the use of mapPartitions. Sometimes, such an operation probably requires a more complicated procedure. This, in the end, makes the method executing the operation needs more than one parameter.
Published:
Have you ever encountered a case where you need to compute the sum of a certain one-item operation? Consider the following example.
Published:
Code profiling is simply used to assess the code performance, including its functions and sub-functions within functions. One of its obvious usage is code optimisation where a developer wants to improve the code efficiency by searching for the bottlenecks in the code.
Published:
Published:
An interesting paper: http://www.phontron.com/paper/oda15ase.pdf
Published:
The theorem of nested cube roots (Ramanujan) states the following.
Published:
There might be a case where we need to perform a certain operation on each data partition. One of the most common examples is the use of mapPartitions. Sometimes, such an operation probably requires a more complicated procedure. This, in the end, makes the method executing the operation needs more than one parameter.
Published:
In Spark, data shuffling simply means data movement. In a single machine with multiple partitions, data shuffling means that data move from one partition to another partition. Meanwhile, in multiple machines, data shuffling can have two kinds of work. The first one is data move from one partition (A) to another partition (B) within the same machine (M1), while the second one is data move from partition B to another partition (C) within different machine (M2). Data in partition C might be moved to another partition within different machine again (M3).
Published:
A statement I encountered a few days ago: “Avoid to use Resilient Distributed Datasets (RDDs) and use Dataframes/Datasets (DFs/DTs) instead, especially in production stage”.
Published:
I made a silly mistake a few days ago - well, yes.
Published:
Published:
In the previous post I wrote about how to derive the Levinson-Durbin recursion.
Published:
I was exploring about the Levinson-Durbin recursion and curious about the formula’s derivation.
Published:
In this post, we’re going to look at how Gradient Boosting algorithm works in a regression problem.
Published:
In the previous post, I mentioned about how collinearity affects the computation of the beta estimators.
Published:
In simple terms, we could define collinearity as a condition where two variables are highly correlated (positively / negatively). When there are more than two variables, it’s sometimes referred as multicollinearity.
Published:
The concept of window function in Spark is pretty interesting. One of its primary usage is calculating cumulative values. Here’s a simple example.
Published:
Recently I played with a simple Spark Streaming application. Precisely, I investigated the behavior of repartitioning on different level of input data streams. For instance, we have two input data streams, such as linesDStream and wordsDStream. The question is, is the repartitioning result different if I repartition after linesDStream and after wordsDStream?
Published:
A few days ago I did a little exploration on Spark’s groupBy behavior. Precisely, I wanted to see whether the order of the data was still preserved when applying groupBy on a repartitioned dataframe.
Published:
A statement I encountered a few days ago: “Avoid to use Resilient Distributed Datasets (RDDs) and use Dataframes/Datasets (DFs/DTs) instead, especially in production stage”.
Published:
Spark has two types of partitioning. The first one is coalesce, while the second one is repartition.
Published:
A few days ago I came across a case where a module needs access to the resources directory.
Published:
Yesterday I came across an interesting Math paper discussing about the Riemann hypothesis. Regarding the concept itself, there’s lots of maths but I think I enjoyed the reading. Frankly speaking, although mathematics is one of my favourite subjects, I’ve been rarely playing with it (esp. pure maths) since I got acquainted with AI and big data engineering world. Now I think it’s just fine to play with it again. Just for fun.
Published:
One of the evaluation metrics that is often optimised is ROC-AUC. In this post, we’re going to discuss how an ROC curve is created.
Published:
The problem is simple. How to find the best threshold from an ROC and PR curve that maximise a certain binary classification metric?
Published:
This article is about a brief overview of how to store several of the most recent log files in PySpark logging.
Published:
Published:
The best way to try new technologies without having clutter? Docker.
Published:
It’s quite bothering when reading a publication that only provides a “statistically significant” result without telling much about the analysis prior to conducting the experiment.
Published:
Applying central moment functions in Spark might be tricky, especially for skewness and kurtosis.
Published:
A few days ago I came across a case where a module needs access to the resources directory.
Published:
There are several ways of removing duplicate rows in Spark. Two of them are by using distinct() and dropDuplicates(). The former lets us to remove rows with the same values on all the columns. Meanwhile, the latter lets us to remove rows with the same values on multiple selected columns.
Published:
A few days ago I did a small experiment with Airflow. To be precise, scheduling Airflow to run a Spark job via spark-submit to a standalone cluster. I have actually mentioned briefly about how to create a DAG and Operators in the previous post.
Published:
I was thinking of the following case.
Published:
Kerberos is simply a “ticket-based” authentication protocol. It enhances the security approach used by password-based authentication protocol. Since there might be a possibility for tappers to take over the password, Kerberos mitigates this by leveraging a ticket (how it is generated is explained below) that ideally should only be known by the client and the service.
Published:
Basically, code obfuscation is a technique used to modify the source code so that it becomes difficult to understand but remains fully functional. The main objective is to protect intellectual properties and prevent hackers from reverse engineering a proprietary source code.
Published:
I encountered an intriguing result when joining a dataframe with itself (self-join). As you might have already known, one of the problems occurred when doing a self-join relates to duplicated column names. Because of this duplication, there’s an ambiguity when we do operations requiring us to provide the column names.
Published:
Published:
Livy offers a REST interface that is used to interact with Spark cluster. It provides two general approaches for job submission and monitoring.
Published:
Primary purpose:
Published:
In Spark, data shuffling simply means data movement. In a single machine with multiple partitions, data shuffling means that data move from one partition to another partition. Meanwhile, in multiple machines, data shuffling can have two kinds of work. The first one is data move from one partition (A) to another partition (B) within the same machine (M1), while the second one is data move from partition B to another partition (C) within different machine (M2). Data in partition C might be moved to another partition within different machine again (M3).
Published:
Have you ever encountered a case where you need to compute the sum of a certain one-item operation? Consider the following example.
Published:
Kalman filter is an iterative mathematical process applied on consecutive data inputs to quickly estimate the true value (position, velocity, weight, temperature, etc) of the object being measured, when the measured values contain random error or uncertainty.
Published:
Have you ever wondered how the size of a dataframe can be discovered? Perhaps it sounds not so fancy thing to know, yet I think there are certain cases requiring us to have pre-knowledge of the size of our dataframe. One of them is when we want to apply broadcast operation. As you might’ve already knownn, broadcasting requires the dataframe to be small enough to fit in memory in each executor. This implicitly means that we should know about the size of the dataframe beforehand in order for broadcasting to be applied successfully. Just FYI, broadcasting enables us to configure the maximum size of a dataframe that can be pushed into each executor. Precisely, this maximum size can be configured via spark.conf.set(“spark.sql.autoBroadcastJoinThreshold”, MAX_SIZE).
Published:
Have you ever wondered how the size of a dataframe can be discovered? Perhaps it sounds not so fancy thing to know, yet I think there are certain cases requiring us to have pre-knowledge of the size of our dataframe. One of them is when we want to apply broadcast operation. As you might’ve already knownn, broadcasting requires the dataframe to be small enough to fit in memory in each executor. This implicitly means that we should know about the size of the dataframe beforehand in order for broadcasting to be applied successfully. Just FYI, broadcasting enables us to configure the maximum size of a dataframe that can be pushed into each executor. Precisely, this maximum size can be configured via spark.conf.set(“spark.sql.autoBroadcastJoinThreshold”, MAX_SIZE).
Published:
Applying central moment functions in Spark might be tricky, especially for skewness and kurtosis.
Published:
First article in 2019.
Published:
Applying central moment functions in Spark might be tricky, especially for skewness and kurtosis.
Published:
A few days ago I came across a case where a module needs access to the resources directory.
Published:
Below is a script for running spark via spark-submit (local mode) that utilizes logging.
Published:
There are several ways of removing duplicate rows in Spark. Two of them are by using distinct() and dropDuplicates(). The former lets us to remove rows with the same values on all the columns. Meanwhile, the latter lets us to remove rows with the same values on multiple selected columns.
Published:
Woe & information value (IV) are used as a framework for attribute relevance analysis. WoE and IV can be utilised independently since each of them play different roles.
Published:
In the previous post, I wrote about how to make LIME run in pseudo-distributed mode with PySpark UDF.
Published:
The initial question that popped up in my mind was how to make LIME performs faster. This should be useful enough when the data to explain is big enough.
Published:
A few days back I tried to set up a spark standalone cluster in my own machine with the following specification: two workers (balanced cores) within a single node.
Published:
Livy offers a REST interface that is used to interact with Spark cluster. It provides two general approaches for job submission and monitoring.
Published:
A few days back I tried to submit a Spark job to a Livy server deployed via local mode. The procedure was straightforward since the only thing to do was to specify the job file along with the configuration parameters (like what we do when using spark-submit directly).
Published:
A few days ago I came across a case where I needed to define a dataframe’s column name with a special character, that is a dot (‘.’). Take a look at thee following schema example.
Published:
A nested column is basically just a column with one or more sub-columns. Take a look at the following example.
Published:
According to the code base, the driver status tracking feature is only implemented for standalone cluster manager. However, based on this reference, we could also poll the driver status for mesos and kubernetes (cluster deploy mode). Additionally, such a feature is also possible for YARN.
Published:
I was thinking of the following case.
Published:
Whenever we call dataframe.writeStream.start() in structured streaming, Spark creates a new stream that reads from a data source (specified by dataframe.readStream). The data passed through the stream is then processed (if needed) and sinked to a certain location.
Published:
Application monitoring is critically important, especially when we encounter performance issues. In Spark, one way to monitor a Spark application is via Spark UI. The problem is, this Spark UI can only be accessed when the application is running.
Published:
There might be a case where we need to perform a certain operation on each data partition. One of the most common examples is the use of mapPartitions. Sometimes, such an operation probably requires a more complicated procedure. This, in the end, makes the method executing the operation needs more than one parameter.
Published:
I was curious about how checkpoint files in Spark structured streaming looked like. To introduce the basic concept, checkpointing simply denotes the progress information of streaming process. This checkpoint files are usually used for failure recovery. More detail explanation can be found here.
Published:
Have you ever encountered a case where you need to compute the sum of a certain one-item operation? Consider the following example.
Published:
Code profiling is simply used to assess the code performance, including its functions and sub-functions within functions. One of its obvious usage is code optimisation where a developer wants to improve the code efficiency by searching for the bottlenecks in the code.
Published:
Unioning two dataframes after joining them with left_anti? Well, seems like a straightforward approach. However, recently I encountered a case where join operation might shift the location of the join key in the resulting dataframe. This, unfortunately, makes the dataframe’s merging result inconsistent in terms of the data in each attribute.
Published:
I encountered an intriguing result when joining a dataframe with itself (self-join). As you might have already known, one of the problems occurred when doing a self-join relates to duplicated column names. Because of this duplication, there’s an ambiguity when we do operations requiring us to provide the column names.
Published:
An intriguing question popped into my mind. After unioning several dataframes, how many partitions the resulting dataframe will have?
Published:
Have you ever wondered how the size of a dataframe can be discovered? Perhaps it sounds not so fancy thing to know, yet I think there are certain cases requiring us to have pre-knowledge of the size of our dataframe. One of them is when we want to apply broadcast operation. As you might’ve already knownn, broadcasting requires the dataframe to be small enough to fit in memory in each executor. This implicitly means that we should know about the size of the dataframe beforehand in order for broadcasting to be applied successfully. Just FYI, broadcasting enables us to configure the maximum size of a dataframe that can be pushed into each executor. Precisely, this maximum size can be configured via spark.conf.set(“spark.sql.autoBroadcastJoinThreshold”, MAX_SIZE).
Published:
In Spark, data shuffling simply means data movement. In a single machine with multiple partitions, data shuffling means that data move from one partition to another partition. Meanwhile, in multiple machines, data shuffling can have two kinds of work. The first one is data move from one partition (A) to another partition (B) within the same machine (M1), while the second one is data move from partition B to another partition (C) within different machine (M2). Data in partition C might be moved to another partition within different machine again (M3).
Published:
The concept of window function in Spark is pretty interesting. One of its primary usage is calculating cumulative values. Here’s a simple example.
Published:
Parquet is a file format with columnar style. Columnar style means that we don’t store the content of each row of the data. Here’s a simple example.
Published:
One of the characteristics of Spark that makes me interested to explore this framework further is its lazy evaluation approach. Simply put, Spark won’t execute the transformation until an action is called. I think it’s logical since when we only specify the transformation plan and don’t ask it to execute the plan, why it needs to force itself to do the computation on the data? In addition, by implementing this lazy evaluation approach, Spark might be able to optimize the logical plan. The task of making the query to be more efficient manually might be reduced significantly. Cool, right?
Published:
Spark functions (UDFs) are simply functions created to overcome speed performance problem when you want to process a dataframe. It’d be useful when your Python functions were so slow in processing a dataframe in large scale. When you use a Python function, it will process the dataframe with one-row-at-a-time manner, meaning that the process would be executed sequentially. Meanwhile, if you use a Spark UDF, Spark will distribute the dataframe and the Spark UDF to the provided executors. Hence, the dataframe processing would be executed in parallel. For more information about Spark UDF, please take a look at this post.
Published:
A few days ago I did a little exploration on Spark’s groupBy behavior. Precisely, I wanted to see whether the order of the data was still preserved when applying groupBy on a repartitioned dataframe.
Published:
A statement I encountered a few days ago: “Avoid to use Resilient Distributed Datasets (RDDs) and use Dataframes/Datasets (DFs/DTs) instead, especially in production stage”.
Published:
I was implementing a paper related to balanced random forest (BRF). Just FYI, a BRF consists of some decision trees where each tree receives instances with a ratio of 1:1 for minority and majority class. A BRF also uses m features selected randomly to determine the best split.
Published:
Spark has two types of partitioning. The first one is coalesce, while the second one is repartition.
Published:
A few days ago I conducted a little experiment on Spark’s RDD operations. One of them was foreach operation (included as an action). Simply, this operation is applied to each rows in the RDD and the kind of operation applied is specified via a certain function. Here’s a simple example:
Published:
I made a silly mistake a few days ago - well, yes.
Published:
This article is about a brief overview of how to store several of the most recent log files in PySpark logging.
Published:
Basically, Metabase’s SparkSQL only allows users to access data in the Hive warehouse. In other words, the data must be in Hive table format to be able to be loaded.
Published:
The problem is really simple. After equi-joining (inner) two dataframes, a certain operation is applied to each partition. Precisely, such an operation can be accomplished by the following code:
Published:
Recently I played with a simple Spark Streaming application. Precisely, I investigated the behavior of repartitioning on different level of input data streams. For instance, we have two input data streams, such as linesDStream and wordsDStream. The question is, is the repartitioning result different if I repartition after linesDStream and after wordsDStream?
Published:
Basically, Metabase’s SparkSQL only allows users to access data in the Hive warehouse. In other words, the data must be in Hive table format to be able to be loaded.
Published:
To discuss about this stack frame, we’ll see from Assembly language point of view.
Published:
Published:
A few days back I tried to set up a spark standalone cluster in my own machine with the following specification: two workers (balanced cores) within a single node.
Published:
Suppose we conduct K experiments on a kind of measurement. On each experiment, we take N observations. In other words, we’ll have N * K data at the end.
Published:
Kalman filter is an iterative mathematical process applied on consecutive data inputs to quickly estimate the true value (position, velocity, weight, temperature, etc) of the object being measured, when the measured values contain random error or uncertainty.
Published:
Kolmogorov-Smirnov (KS) test is a non-parametric test for the equality of probability distributions.
Published:
Published:
One sample z-test is used to examine whether the difference between a population mean and a certain value is significant.
Published:
White noise series has the following properties:
Published:
In the previous post, I mentioned about the general formula of the H statistic is the following (Source: Wikipedia - Kruskal–Wallis one-way analysis of variance):
Published:
The Kruskal-Wallis test is a non-parametric statistical test that is used to evaluate whether the medians of two or more groups are different. Since the test is non-parametric, it doesn’t assume that the data comes from a particular distribution.
Published:
Imagine that you have some data x1, x2, x3, ..., xn originating from an unknown continuous distribution f. You’d like to estimate f.
Published:
In the previous post, we discuss about the implementation of Kalman filter for static state (the true value of the object’s states are constant over time). In addition, the Kalman filter algorithm is applied to estimate single true value.
Published:
Kalman filter is an iterative mathematical process applied on consecutive data inputs to quickly estimate the true value (position, velocity, weight, temperature, etc) of the object being measured, when the measured values contain random error or uncertainty.
Published:
It’s quite bothering when reading a publication that only provides a “statistically significant” result without telling much about the analysis prior to conducting the experiment.
Published:
“If you torture the data long enough, it will confess to anything” - Ronald Coase.
Published:
As the name suggests, moment generating function (MGF) provides a function that generates moments, such as E[X], E[X^2], E[X^3], and so forth.
Published:
Applying central moment functions in Spark might be tricky, especially for skewness and kurtosis.
Published:
One sample z-test is used to examine whether the difference between a population mean and a certain value is significant.
Published:
In the previous post, I mentioned about the basic concept of maximum likelihood estimation (MLE). Please visit the post if you need a refresher.
Published:
If in the probability context we state that P(x1, x2, ..., xn | params) means the probability of getting a set of observations x1, x2, …, and xn given the distribution parameters, then in the likelihood context we get the following.
Published:
In the previous post, I mentioned about the basic concept of two-sample Kolmogorov-Smirnov (KS) test and its implementation in Spark (Scala API).
Published:
Suppose we conduct K experiments on a kind of measurement. On each experiment, we take N observations. In other words, we’ll have N * K data at the end.
Published:
I was thinking of the following case.
Published:
I made a silly mistake a few days ago - well, yes.
Published:
Basically, you can presume Kafka as a messaging system. When an application sends a message to another application, one thing they need to do is to specify how to send the message. The most obvious use case in using a messaging system, in my opinion, is when we’re dealing with big data. For instance, a sender application shares a large amount of data that need to be processed by a receiver application. However, the processing rate by the receiver is lower than the sending rate. Consequently, the receiver might be overloaded since it’s unable to receive messages anymore while the processing is running. Although we’re using distributed receivers, we still have to tell the sender about which receiver node it should send the message to.
Published:
In the previous post, I wrote about how to perform pandas groupBy operation on a large dataset in streaming way. The main problem being addressed is optimum memory consumption since the data size might be extremely large.
Published:
I came across an article about how to perform groupBy operation for large dataset. Long story short, the author proposes an approach called streaming groupBy where the dataset is divided into chunks and the groupBy operation is applied to each chunk. This approach is implemented with pandas.
Published:
Whenever we call dataframe.writeStream.start() in structured streaming, Spark creates a new stream that reads from a data source (specified by dataframe.readStream). The data passed through the stream is then processed (if needed) and sinked to a certain location.
Published:
I was curious about how checkpoint files in Spark structured streaming looked like. To introduce the basic concept, checkpointing simply denotes the progress information of streaming process. This checkpoint files are usually used for failure recovery. More detail explanation can be found here.
Published:
Published:
Published:
Published:
LIME is a python library used to explain predictions from any machine learning classifier.
Published:
A few days ago I did a small experiment with Airflow. To be precise, scheduling Airflow to run a Spark job via spark-submit to a standalone cluster. I have actually mentioned briefly about how to create a DAG and Operators in the previous post.
Published:
The problem is simple. How to find the best threshold from an ROC and PR curve that maximise a certain binary classification metric?
Published:
White noise series has the following properties:
Published:
In the previous post I wrote about how to derive the Levinson-Durbin recursion.
Published:
I was exploring about the Levinson-Durbin recursion and curious about the formula’s derivation.
Published:
Just wanted to confirm whether the Kafka consumers were aware of new topic’s partitions.
Published:
One of the techniques in hyperparameter tuning is called Bayesian Optimization. It selects the next hyperparameter to evaluate based on the previous trials.
Published:
Let’s take a look at the problem statement.
Published:
One of the evaluation metrics that is often optimised is ROC-AUC. In this post, we’re going to discuss how an ROC curve is created.
Published:
Published:
The initial question that popped up in my mind was how to make LIME performs faster. This should be useful enough when the data to explain is big enough.
Published:
In the previous post, we discuss about the implementation of Kalman filter for static state (the true value of the object’s states are constant over time). In addition, the Kalman filter algorithm is applied to estimate single true value.
Published:
Kalman filter is an iterative mathematical process applied on consecutive data inputs to quickly estimate the true value (position, velocity, weight, temperature, etc) of the object being measured, when the measured values contain random error or uncertainty.
Published:
Unioning two dataframes after joining them with left_anti? Well, seems like a straightforward approach. However, recently I encountered a case where join operation might shift the location of the join key in the resulting dataframe. This, unfortunately, makes the dataframe’s merging result inconsistent in terms of the data in each attribute.
Published:
An intriguing question popped into my mind. After unioning several dataframes, how many partitions the resulting dataframe will have?
Published:
If you read my previous article titled Union Operation After Left-anti Join Might Result in Inconsistent Attributes Data, it was shown that the attributes data was inconsistent when combining two data frames after inner-join. According to the article, the solution is really simple. We just need to reorder the attributes order by using select command. Here’s a simple example.
Published:
Recently I was exploring ways of adding a unique row ID column to a dataframe. The requirement is simple: “the row ID should strictly increase with difference of one and the data order is not modified”.
Published:
In the previous post, I mentioned about how collinearity affects the computation of the beta estimators.
Published:
Let’s take a look at the problem statement.
Published:
Last time I wrote about the infinite products representation for pi that is regarded as the Wallis’ product for pi.
Published:
In the previous post I mentioned about how to demonstrate the Wallis product for pi by starting from the powered sine integration.
Published:
The Wallis’ infinite product for pi states the following.
Published:
I was experimenting with the weight of evidence (WoE) encoding for continuous data. The preparation is quite different from categorical data in terms of binning characteristics.
Published:
Woe & information value (IV) are used as a framework for attribute relevance analysis. WoE and IV can be utilised independently since each of them play different roles.
Published:
Published:
White noise series has the following properties:
Published:
There are several ways of removing duplicate rows in Spark. Two of them are by using distinct() and dropDuplicates(). The former lets us to remove rows with the same values on all the columns. Meanwhile, the latter lets us to remove rows with the same values on multiple selected columns.
Published:
The concept of window function in Spark is pretty interesting. One of its primary usage is calculating cumulative values. Here’s a simple example.
Published:
A few days back I tried to set up a spark standalone cluster in my own machine with the following specification: two workers (balanced cores) within a single node.
Published:
A few days ago I did a small experiment with Airflow. To be precise, scheduling Airflow to run a Spark job via spark-submit to a standalone cluster. I have actually mentioned briefly about how to create a DAG and Operators in the previous post.
Published:
Airflow is basically a workflow management system. When we’re talking about “workflow”, we’re referring to a sequence of tasks that needs to be performed to accomplish a certain goal. A simple example would be related to an ordinary ETL job, such as fetching data from data sources, transforming the data into certain formats which in accordance with the requirements, and then storing the transformed data to a data warehouse.
Published:
Let’s use a simple example data to demonstrate how XGBoost algorithm works on a classification problem.
Published:
One sample z-test is used to examine whether the difference between a population mean and a certain value is significant.
Published:
It’s quite bothering when reading a publication that only provides a “statistically significant” result without telling much about the analysis prior to conducting the experiment.
Published:
One sample z-test is used to examine whether the difference between a population mean and a certain value is significant.
Published:
Yesterday I came across an interesting Math paper discussing about the Riemann hypothesis. Regarding the concept itself, there’s lots of maths but I think I enjoyed the reading. Frankly speaking, although mathematics is one of my favourite subjects, I’ve been rarely playing with it (esp. pure maths) since I got acquainted with AI and big data engineering world. Now I think it’s just fine to play with it again. Just for fun.