Table of Contents
show
What is data Cleansing?
Simply, it cleans the data.
It looks for corrupted data records and hunts them down. We can apply this whole data cleansing process on a set of data records, data tables, or on an entire database. At times, after a full process of web scraping or data harnessing, you can end up with a lump of dirty data sets as the sole output of the entire hard earned day. What happens next? It affects your decision-making quality as the analysis of dirty data veers you off from the true information. The web also coins this data cleansing process with different names, like data cleaning or data scrubbing. Although both these names are uttered interchangeably still, there is a fine demarcation line between them.
Data cleansing can be termed as a process of removing errors and incongruities from the crawled or scrapped web data which will be used as the source data for analysis but data scrubbing deals with a herd of data quality issues like data filtering, data merging, decoding and translating the source data into a pool of validated data for the purpose of data warehousing.
If anyone looks deep inside the whole concept of ‘big data’, he can infer that it is like a cursed blessing and one of the curses big data lashes the business world with is the ‘dirty data’. The major chunk of the human race, who deal with big data in any way, know the term. If you are a bit less caffeinated soul like me on this issue then, the phrase ‘dirty data’ may not appear that friendly to you.
So, what can be defined as ‘dirty data’?
Simply, a data set can be termed as dirty data if there are impurities in it. That being said, if a data set contains any dummy value, unfilled data field, encrypted or gibberish value for a data field, no unique identifiers or alike anomalies then, it is surely dirty as it will not fulfil the very purpose of data analytics. Consequently, without a scientific and errorless data analysis, big data is nothing but just a dead weight on the process of your business strategy planning.
So, if you need to harness the information hiding beneath the big data then, you need to have a cleansed or scrubbed data set as a pristine input to your data analysis system, else the whole purpose of analysing that data chunk will succumb to nothing. So, if you are looking for your business’s future in big data then it is absolutely necessary to get clean data sets through web crawling or web scraping.
What to know before executing data cleansing process?
Before intruding further into data cleansing, it is pretty important to have a cognizance about the purpose and the standards of data cleansing and this segment of the entire process can be encased under ‘data quality’ tag. Moreover, DQA or data quality assurance is a benchmark process for checking the health of warehoused data chunks before put to use.
So, what are the standards of data quality? Believe me, there are many and each of them is more important than the rest. There is a herd of data churning technologies which populate the data quality factors. They are:
- Data accuracy: It deals with the degree of conformity which decides whether a particular data set beats the standard of true value or not.
- Data validity: it defines whether a particular data set falls within the valid region of data characteristics or not. This particular data quality factor houses an array of data constraint factors like data type, mandatory, unique, foreign intrusion with the other alike fields.
- Completeness factor: This defines the completeness of the requested data sets and it is the single most threatening factor against data cleansing. Simply, it is next to impossible to clean an incomplete data set.
- Reliability: It defines how reliable is the data? That being said, will it be able to prove its character to other databases?
Admittedly, there are other data quality factors like data consistency, data uniformity, data integrity and others, except this list and they also belong to the same importance level.
How to clean data?
Typically, there are five steps.
-
Data analysis
This step includes the entire detection part. The key focus of this step is to locate and analyse the errors and inconsistencies of the data which is under the scanner. There are various approaches to analyse the data. It starts with a manual inspection to the data sampling part and ends with a complete analysis of metadata information, other allied data properties and data detection quality issues.
-
Transformation workflow with the data mapping rules
Here, data transformation or cleansing workflow depends upon various factors like the degree of dirtiness, number of data sources and their basic differences. Cleansing processes like schema translation to map sources, cleaning of single source instances with other multi source instances like cleaning duplicate entries and alike. Moreover, the entire process of data transformation and data integration for warehousing should define the entire process of ETL (extraction, transformation and loading).
-
Correctness verification
This step measures the effectiveness of the transformation workflow and evaluates the same. Here, multiple iterations of data analysis are needed to re-detect any existing anomalies and to clean them.
-
Complete transformation
It is the complete execution of the data transformation process. In this step, the entire ETL process transforms the data, loads the transformed data to a data warehouse and finally, completes the process by refreshing the data warehouse.
-
Return flow of clean data
This last step ensures that the cleaned data has replaced dirty data at the original source. This is extremely important for the future data extractions from the same data source. As the data extraction processes are iterative in their nature so, to start with pristine data sets at the source, every single time, is a bottom line requirement.
The future of every enterprise depends on the accuracy of this data cleansing process as big data may have gifted us with the availability of the petabytes of data, but the quality of information or the groundbreaking insights hiding beneath these data layers depends upon that process which is churning and cleaning it to gold
