Skip to main content

Best Practice To Trim Before Removing Duplicates or Merging In Power Query Editor

In last week’s blog, I wrote and did a video about how to remove duplicate records and keep the most recent entry as long as a date column was part of the data source.  I came across the scenario while giving training on Power BI with my company Pragmatic Works.  See the video below:

 

 

This week, while doing another two-day training I came across a different scenario from a follow-up conversation from day 1.  I had explained how to remove duplicate records and one of the students started working on a Power BI project she has for her company.  On day 2 the student informed me that her remove duplicates step was not working.  I said that is odd and I asked to see the data.  In one of her table visuals, I could see that it appeared that a few of the records had duplicates based on the name column.  After further investigation though, we figured out the culprit. 

 


She had done all the steps correctly, but it was a data integrity issue.  In her data source, the person in charge of entering the data had mistakenly typed a space for a few of the records before entering the name value.  Thus, when Power BI looked for a duplicate in the name column it did its job correctly because “Jamie” is unique compared to “ Jamie”.  That leading white space was the culprit.  Not only did it affect her Remove Duplicates step, but it would also have affected a merging of two queries if she had included that as part of her report. 

 


For this week’s video, I want to show how to use the Trim function in the Power Query Editor as a best practice to avoid these unwanted outcomes for any future merges or removing of duplicates.



Labels:

Comments

Post a Comment

Popular posts from this blog

The Power of CALCULATE in Power BI

Last week in my blog and video I discussed how Power BI could have made my life as a teacher much easier.  It is not that Power BI can do things that I could not on my own.  It is the fact that Power BI could have saved me so much time.   This led me to some thoughts and questions.   How could I have impacted the students more in the classroom with this extra time?  Would teachers who were not analyzing student data due to time constraints now start?  How can I make the data tell my students a story that gives them an accurate picture of their progress?   That last question is what leads to this week’s blog.   Many times the metrics (which is just a fancy word for "numbers") of what we want to show are easy to do by hand with a basic math background.  Although easy, it can be very time consuming when dealing with different groups of data: like 6 periods of students instead of 1 period. When we want to do these calculations quickl...
7 comments
Read more

Relating "Related Tables" to Baseball because I Miss Sports

I miss sports. In particular, I miss baseball. Between learning more Power BI functions and the ins-and-outs of DAX, I've turned to Netflix to fill the deep caverns left in my soul since baseball season has been postponed. And as a result, I've thought more about tigers and big cats more than I ever have in my life. I know ALL about Carol Baskins and am fully on board for a spin-off centering on locating her lost husband. I've googled "is it really legal to own a tiger in a residential area?" Without baseball in April, I am barely hanging in there (kinda like Joe Exotic's eyebrow ring). So, I am filling the sports-sized hole by using baseball stats in Power BI to demonstrate pulling data from multiple tables and consolidating it into one table.  Some of the data we want to consolidate also has to have some aggregations (which is fancy for "calculations") performed on it.  In this demo I will attempt to break down what is really going on ...
Post a Comment
Read more

Power BI: Personalize Visuals Feature

    Have you used the personalize visuals feature yet in Power BI?   If not, then you could be missing out.   The Personalize Visuals feature allows anyone you are sharing your with after published to the Power BI service to make changes to your visuals.   Have no fear though!   When they change the visuals it will not override your original report.   This feature is great if you have ever had to respond to a request from your shared users that they would like the report to be modified to their liking.   With this simple feature, the end-user can change the visualization type, change what measures or fields that are brought into the visual, and then simply make a personal bookmark after they make the changes.   Again, this will not change the initial report you published and that others have access to.   This will only make a new view of the report for the user who personalized the visual.   If you decide to use this feature...
2 comments
Read more