Posts

Showing posts with the label data ingestion

Split Datasets

Image
 The Objective of this article is to transform data set from row to column using explode() method. The scope of this article is to understand how to  unnest or explode a data set using parallel processing framework Pyspark and Python native library- Pandas . Dataset looks like as below: dept,name 10,vivek#ruby#aniket 20,rahul#john#amy 30,shankar#jagdish 40, 50,yug#alex#alexa Pandas explode() import pandas as pd pan_df=pd.read_csv(r'explode.csv') df_exp=pan_df.assign(name=pan_df['name'].str.split('#')).explode('name') df_exp Output: Dataset is transformed successfully and we are able to create new rows from nested dataset. Pandas way of explode is simple, crisp and straight forward unless the dataset is complex. In next section of this article we will cover PySpark way of exploding or unnesting dataset. PySpark explode() Import libraries and Connect to Spark from pyspark import SparkContext,SparkConf import pyspark from pyspark.sql import SparkSes
Post a Comment
Read more

Split Datasets

Image
 The Objective of this article is to transform data set from row to column using explode() method. The scope of this article is to understand how to  unnest or explode a data set using parallel processing framework Pyspark and Python native library- Pandas . Dataset looks like as below: dept,name 10,vivek#ruby#aniket 20,rahul#john#amy 30,shankar#jagdish 40, 50,yug#alex#alexa Pandas explode() import pandas as pd pan_df=pd.read_csv(r'explode.csv') df_exp=pan_df.assign(name=pan_df['name'].str.split('#')).explode('name') df_exp Output: Dataset is transformed successfully and we are able to create new rows from nested dataset. Pandas way of explode is simple, crisp and straight forward unless the dataset is complex. In next section of this article we will cover PySpark way of exploding or unnesting dataset. PySpark explode() Import libraries and Connect to Spark from pyspark import SparkContext,SparkConf import pyspark from pyspark.sql import SparkSes
Post a Comment
Read more

pySQL

Image
 The Objective of this article is to understand how to perform data manipulation on pandas DataFrames using SQL with pandasql library.  What is pandasql? As per documentation pandasql allows us to query pandas DataFrames using SQL syntax. Installation of pandasql Library can be installed using below two methods, both of them uses PIP installation: Using Terminal pip install -U pandasql Using Jupyter Notebooks !pip install -U pandasql Use Case The main function used in pandasql is sqldf. sqldf accepts two arguments:  SQL query  Session environment variables ( globals() and locals() ) session environment variables is optional and handled by python itself, even if we do not provide the session variables, we still can achieve the goal of converting pandas DataFrame to perform SQL querying. Import Necessary Libraries from pandasql import sqldf import pandas as pd Import Data Sets emp_df=pd.read_csv(r'D:\python_coding\pandas_practice\emp.csv') emp_df.head(10) dept_df=pd.read_csv(r&
Post a Comment
Read more

Spark Window Functions

Image
 The objective of this article is to understand Pyspark Window functions. The blog will do a comparative study of Pyspark window functions and Relational DB systems, Oracle Database, analytical functions. Spark Window functions operate on a group of rows (like frame, partition) and return a single value for every input row.  To perform an operation on a group first, we need to partition the data using Window.partitionBy() , and for row number and rank function we need to additionally order by on partition data using orderBy()  clause. Connect to Spark import pyspark from pyspark.sql import SparkSession print('modules imported') spark=SparkSession.builder.appName('Spark_window_functions').getOrCreate() Load Dataset emp_df=spark.read.csv(r'emp.csv',header=True,inferSchema=True) emp_df.show(10) Import necessary Libraries from pyspark.sql.window import Window from pyspark.sql.functions import col, row_number, rank, dense_rank from pyspark.sql import functions as
Post a Comment
Read more

Append Datasets

Image
 In the Data universe, Joins and Unions are the most critical and frequently performed operations. In my experience, almost every other operation is either a join or a union. As joins are inevitable so do unions. In previous article we have covered how joins work in Pandas. Link to article:  https://letscodewithvivek.blogspot.com/2021/12/python-joins.html The scope of this article is to understand about how  concat()  methods helps us achieve the union of data frames. concat() Concatenate or concat pandas objects along a particular axis with optional set logic along the other axes. create two data frames to understand how concat method works. concat data frames on axis=0, default operation (union) import pandas as pd df1 = pd.DataFrame('Name': ['Vivek', 'Amy', 'Vishakha', 'Alice', 'Ayoung'], 'subject_id':['sub1','sub2','sub4','sub6','sub5'], 'Marks_scored':[98,90,87,69,
Post a Comment
Read more

Popular posts from this blog

Split Datasets

Image
 The Objective of this article is to transform data set from row to column using explode() method. The scope of this article is to understand how to  unnest or explode a data set using parallel processing framework Pyspark and Python native library- Pandas . Dataset looks like as below: dept,name 10,vivek#ruby#aniket 20,rahul#john#amy 30,shankar#jagdish 40, 50,yug#alex#alexa Pandas explode() import pandas as pd pan_df=pd.read_csv(r'explode.csv') df_exp=pan_df.assign(name=pan_df['name'].str.split('#')).explode('name') df_exp Output: Dataset is transformed successfully and we are able to create new rows from nested dataset. Pandas way of explode is simple, crisp and straight forward unless the dataset is complex. In next section of this article we will cover PySpark way of exploding or unnesting dataset. PySpark explode() Import libraries and Connect to Spark from pyspark import SparkContext,SparkConf import pyspark from pyspark.sql import SparkSes
Read more

pySQL

Image
 The Objective of this article is to understand how to perform data manipulation on pandas DataFrames using SQL with pandasql library.  What is pandasql? As per documentation pandasql allows us to query pandas DataFrames using SQL syntax. Installation of pandasql Library can be installed using below two methods, both of them uses PIP installation: Using Terminal pip install -U pandasql Using Jupyter Notebooks !pip install -U pandasql Use Case The main function used in pandasql is sqldf. sqldf accepts two arguments:  SQL query  Session environment variables ( globals() and locals() ) session environment variables is optional and handled by python itself, even if we do not provide the session variables, we still can achieve the goal of converting pandas DataFrame to perform SQL querying. Import Necessary Libraries from pandasql import sqldf import pandas as pd Import Data Sets emp_df=pd.read_csv(r'D:\python_coding\pandas_practice\emp.csv') emp_df.head(10) dept_df=pd.read_csv(r&
Read more

Can Julia compete PySpark? A Data Comparison

Image
Creators of Julia language claims Julia to be very fast, performance wise as it does not follow the two language theory like Python, it is a compiled language where as Python is an amalgamation of both compilation and interpretation. It would be interesting to dig deep to understand about how both of these languages behave behind the scenes but the objective of this blog is not to get into the theoretical details of the differences.  As a Data Engineer my innate behavior is to understand how Julia behaves when it is bombed with GBs or TBs of data sets. As I am talking about GBs or TBs of data sets so obviously I can not straight away compare Python with Julia or even the rich Pandas library, as all of us know the processing will never complete as python is quiet slow. So the scope of this blog is to draw parallels between Julia and PySpark, I know for some this is unfair but pardon me. Inspiration behind the blog is the Twitter podcast on Julia that took place in Jan 2022.  #Note: I ha
Read more