How to remove or prevent duplicates in Google Photos
Google Photos tries to avoid exact duplicates, by allotting a hash code to identify each file. This code changes when the file is edited or moved across devices. When different devices are used, or photos have been edited or metadata changed, (near) duplicates may be uploaded. A different title/path does not trigger a new upload.
Also important: the Google Photos app on mobile devices not only shows photos/videos uploaded to Google Photos, but also photos in folders on the device. That may under certain circumstances (for example moving photos to another folder, adding photos to the device) lead to duplicates in the Photos tab. To see what is really uploaded to Google Photos you should use the web app https://photos.google.com
When we get new info on how to avoid/remove duplicates we will add it here.
Reasons for getting duplicates
If you uploaded using Picasa, a new upload to Google Photos created duplicates of all photos that were modified in Picasa, for example edited, date & time changed, tags added, etc. Renaming a photo did not result in duplicates.
Uploading from multiple devices, for example, a smartphone and computer. EXACT copies of photos will not be duplicated, but VIDEOS will be duplicated.
When switching to a new phone, some people reported seeing duplicates. It's not clear why or how this triggers duplicates.
New or reformatted computer may cause duplicates.
It may upload RAW & JPEG or HEIC & JPEG
Many asked for a de-duplication program, but Google says that "it cannot decide which photo the user wants to keep". What would already help is a search option to display duplicates side-by-side so that the user can more easily decide.
Things to check
In order to find out where the duplicates come from you can check a few things:
Check in the info field in the web app https://photos.google.com/ where each copy comes from, for example "Uploaded from a browser", "Uploaded from an Android (or iOS) device", "Shared by Name, saved to your Photos", etc
Check in the "recently added" field (under "Explore" in the web app and under "Search" in the mobile app) whether both are uploaded at the same day or not.
Do both duplicates have the same or different size? Smaller copies may be thumbnails.
Your camera may be set to take both RAW and JPEG
It happens that the same photo is shown twice, especially in People, things, places etc or in search results. Deleting one also deletes the other. You can check by adding a "Description" to one of the duplicates: it's not a real duplicate when the Description is also added to the other photo.
Recommendations to remove duplicates
There is no easy way to remove duplicates in Google Photos (prevention is important!), but this may help a bit:
You can delete duplicates manually, for example in https://photos.google.com/ where they should be next to each other except when date/time was changed.
Duplicates may also appear together in albums created by Google under People, Places and Things, but you have to be careful: sometimes a photo is shown twice. When you add for example something in the Description field", and it also appears in the second photo, there is in fact only one photo, and deleting one will delete both.
You may have to check which ones you can delete:
One photo may be edited, the other not. You need to open the photo and click the edit icon to find out.
Deleting a photo removes it from ALL albums. One photo may be in an album, the other not or in more/different albums. You can see in the info panel to which album(s) a photo is added and add the photo you want to keep to ALL albums.
One photo may have more pixels or more KB or MB (see info of the photo)
It may be easier to delete batches of duplicates in https://photos.google.com/search/_tra_ where they are sorted by upload date (most recent on top). That separates two sets of same photos so that you can select a range to delete.
You can also add a bunch of photos and duplicates to an album (it can have 20.000 photos now), and sort by upload date (Open the album > click "More options" (3-dots ) > click "Sort" (2 arrows) > "Recently added". That will group the photos by upload date, making it possible to select multiple photos and delete them in batch. It is possible to delete photos directly from an album (in web app only). Go to "More options" (3-dots icon), and use "Move to bin".
When you uploaded in the free "High Quality" (or converted to "High Quality" using "Recover storage" in https://photos.google.com/settings) and have all your photos in albums, you could ignore the duplicates in the library, since they do not use your storage. This will change on June 1, 2021. The files uploaded with setting HQ before that date will still not count for storage, but new files will.
Third party software
Duplicate removal in Google Photos:
https://duplicates-google-photos.remikel.fr/
tutorial https://youtu.be/ULalSu3MPRE
See also https://support.google.com/photos/thread/91700588?hl=en&msgid=92347853
https://github.com/mtalcott/google-photos-deduper
https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper
Duplicate removal after download to a computer
Some users suggested programs to delete duplicates on a computer hard disk. See here how to download photos from Google Photos to a computer. When you manage to create a group of photos that could be removed, you can proceed as follows in order to avoid deleting and re-uploading photos (and thus avoiding to replace photos uploaded for free before June 2021 to use storage):
Download ALL photos to a computer. Note that, when using Takeout, you should only download the year albums, like shown here: https://support.google.com/photos/thread/185261667. Downloading other albums will generate more duplicates making this method unusable.
Use a program that can detect duplicates and keep a list of photos that can be removed.
Create a new album on https://photos.google.com/albums
Drag the photos that can be deleted in the open album or use "Select photos" > Select from computer.
The album now contains all these photos. Already backed up photos will not upload again.
Important: this is a critical step: when the duplicate detecting program added or changed something in the metadata of the photos, this method will not work. Try with a few examples, and check as follows:Go to "Recently added" (https://photos.google.com/search/_tra_)
When the photos are "added today" the method does not work, then
select all photos
click on 3-dots > Move to bin
When the photos are NOT shown under "Added today" in "Recently added", continue as explained below
Select all photos in the album
Click 3-dots > Move to bin.
All duplicates should now be deleted.
Programs mentioned by users:
Picasa3 (still used by many) has the option to display duplicates close to each other, thus facilitating manual removal. Tools > Experimental > Show Duplicate Files
https://blogs.systweak.com/11-duplicate-photo-finder-tools-to-delete-duplicate-photos/