Skip to main content

Best Practice To Trim Before Removing Duplicates or Merging In Power Query Editor

In last week’s blog, I wrote and did a video about how to remove duplicate records and keep the most recent entry as long as a date column was part of the data source.  I came across the scenario while giving training on Power BI with my company Pragmatic Works.  See the video below:

 

 

This week, while doing another two-day training I came across a different scenario from a follow-up conversation from day 1.  I had explained how to remove duplicate records and one of the students started working on a Power BI project she has for her company.  On day 2 the student informed me that her remove duplicates step was not working.  I said that is odd and I asked to see the data.  In one of her table visuals, I could see that it appeared that a few of the records had duplicates based on the name column.  After further investigation though, we figured out the culprit. 

 


She had done all the steps correctly, but it was a data integrity issue.  In her data source, the person in charge of entering the data had mistakenly typed a space for a few of the records before entering the name value.  Thus, when Power BI looked for a duplicate in the name column it did its job correctly because “Jamie” is unique compared to “ Jamie”.  That leading white space was the culprit.  Not only did it affect her Remove Duplicates step, but it would also have affected a merging of two queries if she had included that as part of her report. 

 


For this week’s video, I want to show how to use the Trim function in the Power Query Editor as a best practice to avoid these unwanted outcomes for any future merges or removing of duplicates.



Labels:

Comments

Post a Comment

Popular posts from this blog

The Power of CALCULATE in Power BI

Last week in my blog and video I discussed how Power BI could have made my life as a teacher much easier.  It is not that Power BI can do things that I could not on my own.  It is the fact that Power BI could have saved me so much time.   This led me to some thoughts and questions.   How could I have impacted the students more in the classroom with this extra time?  Would teachers who were not analyzing student data due to time constraints now start?  How can I make the data tell my students a story that gives them an accurate picture of their progress?   That last question is what leads to this week’s blog.   Many times the metrics (which is just a fancy word for "numbers") of what we want to show are easy to do by hand with a basic math background.  Although easy, it can be very time consuming when dealing with different groups of data: like 6 periods of students instead of 1 period. When we want to do these calculations quickl...
7 comments
Read more

The Teacher Becomes the Student

My wife asked how my first day was going so I sent her this selfie. Only a picture could capture it.  First, let me tell you that this won’t be your typical Power BI or tech blog that you usually see. Let me set the scene. It was a year and a half ago when I got lucky enough to be introduced to Brian Knight, the CEO of Pragmatic Works over lunch. He had heard from a mutual friend that I was looking to make a career move after teaching Algebra for the past 15 years. Brian started telling me about the company and all the different facets it encompasses. From training, sales, consulting, and app development. Mind you at this time I had never heard of Microsoft Power BI, SQL Server, Microsoft Azure, or have any experience with coding. He told me an opportunity at his company may present itself in the future where I could transfer over my teaching skills into the training side of the above-mentioned programs. I immediately got to work with their On-Demand Learning system to ...
3 comments
Read more

Power Automate: Send Instant Emails From Canvas Apps

 In this blog, I want to dive into Power Automate.  When I first started with the Power Platform I dove deep into Power BI which then made way for Power Apps.  Soon thereafter I was introduced to Power Automate.  I sure wish someone had told me about Power Automate when I was a teacher of middle schoolers.  There were so many mundane daily and weekly tasks that I could have easily automated if I just would have been exposed to the program.   Power Automate is great for taking those routine or repetitive tasks and making them no longer a part of your daily routine. Sure setting up the Power Automate flow takes more time to get created, but once created you then can devote all that extra time you normally would have devoted to the task and now spend them on more mission-critical issues for your job.   In this video, I want to show you how you can send an automated email with just a click of a button.  Since I learned Power Automate after lea...
Post a Comment
Read more