Remove Duplicate Tuples From A List If They Are Exactly The Same Including Order Of Items

I know questions similar to this have been asked many, many times on Stack Overflow, but I need to remove duplicate tuples from a list, but not just if their elements match up, the

Solution 1:

What you need is unique permutations rather than combinations:

y = list(set(itertools.permutations(x,3)))

That is, (1,2,2) and (2,1,2) will be considered as same combination and only one of them will be returned. They are, however, different permutations. Use set() to remove duplicates.

If afterwards you want to sort elements within each tuple and also have the whole list sorted, you can do:

y = [tuple(sorted(q)) for q in y]
y.sort()

Solution 2:

No need to do for loop, combinations gives a generator.

x = [1,1,1,2,2,2,3,3,3,4,4,5]
y = list(set(itertools.combinations(x,3)))

Solution 3:

This will probably do what you want, but it's vast overkill. It's a low-level prototype for a generator that may be added to itertools some day. It's low level to ease re-implementing it in C. Where N is the length of the iterable input, it requires worst-case space O(N) and does at most N*(N-1)//2 element comparisons, regardless of how many anagrams are generated. Both of those are optimal ;-)

You'd use it like so:

>>>x = [1,1,1,2,2,2,3,3,3,4,4,5]>>>for t in anagrams(x, 3):...print(t)
(1, 1, 1)
(1, 1, 2)
(1, 1, 3)
(1, 1, 4)
(1, 1, 5)
(1, 2, 1)
...

There will be no duplicates in the output. Note: this is Python 3 code. It needs a few changes to run under Python 2.

import operator

classENode:
    def__init__(self, initial_index=None):
        self.indices = [initial_index]
        self.current = 0
        self.prev = self.next = self

    defindex(self):
        "Return current index."return self.indices[self.current]

    defunlink(self):
        "Remove self from list."
        self.prev.next = self.next
        self.next.prev = self.prev

    definsert_after(self, x):
        "Insert node x after self."
        x.prev = self
        x.next = self.next
        self.next.prev = x
        self.next = x

    defadvance(self):
        """Advance the current index.

        If we're already at the end, remove self from list.

        .restore() undoes everything .advance() did."""assert self.current < len(self.indices)
        self.current += 1if self.current == len(self.indices):
            self.unlink()

    defrestore(self):
        "Undo what .advance() did."assert self.current <= len(self.indices)
        if self.current == len(self.indices):
            self.prev.insert_after(self)
        self.current -= 1defbuild_equivalence_classes(items, equal):
    ehead = ENode()
    for i, elt inenumerate(items):
        e = ehead.nextwhile e isnot ehead:
            if equal(elt, items[e.indices[0]]):
                # Add (index of) elt to this equivalence class.
                e.indices.append(i)
                break
            e = e.nextelse:
            # elt not equal to anything seen so far:  append# new equivalence class.
            e = ENode(i)
            ehead.prev.insert_after(e)
    return ehead

defanagrams(iterable, count=None, equal=operator.__eq__):
    defperm(i):
        if i:
            e = ehead.nextassert e isnot ehead
            while e isnot ehead:
                result[count - i] = e.index()
                e.advance()
                yieldfrom perm(i-1)
                e.restore()
                e = e.nextelse:
            yieldtuple(items[j] for j in result)

    items = tuple(iterable)
    if count isNone:
        count = len(items)
    if count > len(items):
        return

    ehead = build_equivalence_classes(items, equal)
    result = [None] * count
    yieldfrom perm(count)

Solution 4:

You were really close. Just get permutations, not combinations. Order matters in permutations, and it does not in combinations. Thus (1, 2, 2) is a distinct permutation from (2, 2, 1). However (1, 2, 2) is considered a singular combination of one 1 and two 2s. Therefore (2, 2, 1) is not considered a distinct combination from (1, 2, 2).

You can convert your list y to a set so that you remove duplicates...

import itertools

x = [1,1,1,2,2,2,3,3,3,4,4,5]
y = []

for x in itertools.permutations(x,3):
    y.append(x)
print(set(y))

And voila, you are done. :)

Solution 5:

Using a set should probably work. A set is basically a container that doesn't contain any duplicated elements.

Python also includes a data type for sets. A set is an unordered collection with no duplicate elements. Basic uses include membership testing and eliminating duplicate entries. Set objects also support mathematical operations like union, intersection, difference, and symmetric difference.

import itertools

x = [1,1,1,2,2,2,3,3,3,4,4,5]
y = set()

for x in itertools.combinations(x,3):
    y.add(x)
print(y)