filter() definition simplified? (PY)

Question

I just don't get it, or the syntax either.

Answer 1

The Python filter() function is like a shortcut to writing a for loop.

It is not a replacement for all for loops, but rather a specific kind with an if statement that chooses which items belong or don't belong in a group of items.

You supply the items to filter in its second argument. Filter loops over those items and tests each one according to the conditional function that you supply in the first argument.

Like a for loop, filter passes each item (one at a time) as an argument into the True/False function, then checks the return value. If your function returns false, then filter ignores that item and skips to the next item. If your function returns true, then filter includes that item in its output.

filter(<True/False function>, <iterable items>)

Answer 2

Bob_Li ,

this is what the *python docs* say about `filter` and `generator`:

filter(function, iterable)

> Construct an iterator from those elements of iterable for which function is true. iterable may be either a sequence, a container which supports iteration, or an iterator. If function is None, the identity function is assumed, that is, all elements of iterable that are false are removed.

> Note that filter(function, iterable) is equivalent to the generator expression:
(item for item in iterable if function(item))
if function is not None and

(item for item in iterable if item)
if function is None.

> See itertools.filterfalse() for the complementary function that returns elements of iterable for which function is false.

Answer 3

Lothar
'filter object' does not necessarily mean it is not a generator. See this behavior:

nums = (1,2,3,4,5,6)
res = filter(lambda x: x%2==0, nums)

print(res)
print(next(res))
print(next(res))
print(next(res))
print(next(res))

compare with it's generator expression equivalent:

nums = (1,2,3,4,5,6)
res2 = (i for i in nums if i%2==0)

print(res2)
print(next(res2))
print(next(res2))
print(next(res2))
print(next(res2))

so, is filter a generator? Duck test seems to say yes...
which makes me doubt my previous claim of filter being slower, since they are using the same technology. The source of the claim were a few SO and blog articles.

Answer 4

Bob_Li ,

filter object supports both next() and iter() because it is an *iterator*.

filter objects are lazy, meaning they don't evaluate until you iterate over them.

you can use iter() on a filter object to obtain an iterator (though the filter object itself is already an iterator).

you can also use next() to retrieve the next item from the filter object.

Answer 5

Lothar
I am finding it hard to think of a scenario where you would opt to use itertools.filterfalse..🤔
Anyway, with the optimizations in list comprehension, even the usage of filter is not the most optimal thing to do. List comprehension is faster than filter. And if you don't need a list, a generator expression is faster than a list comprehension.
But as with all things, that's no excuse not to learn about filter. They are easier to write than list comprehensions, specially if the condtion is somewhat complex.
Also, one thing to remember about filter is that since it is a generator, you hit StopIteration when there's nothing more to yield, so you can't just reuse the variable over and over again.

Answer 6

Bob_Li ,

my last post was just a copy of the complete description of filter() function from the python docs, including the hint of `filterfalse()`.

my point was to mention that filter is not `a generater function` in your sample, filter just returns an iterator / filter object.

nums = (1,2,3,4,5,6)
res = filter(lambda x: x%2==0, nums)
print(res) # will return like: <filter object at 0x74c74ee3b0>

Answer 7

Lothar
hmmm... this grilling is actually helpful...👍

Upon opening the python repl and typing
>> help(filter)

I learned that filter is actually a class.
So why? maybe to interact with cpython more deeply to gain performance boost?

So I dug into how generators are implemented in CPython..

https://stackoverflow.com/questions/8389812/how-are-generators-and-coroutines-implemented-in-cpython

and the link to genobject.c
https://github.com/python/cpython/blob/main/Objects%2Fgenobject.c

I went into this expecting a simple generator function for filter, but Python is actually implementing it at a lower level to produce an iterable.

and no, I can't even pretend to understand those C codes.

Answer 8

Bob_Li ,

thanks for this information.

Answer 9

Thanks

Answer 10

nums = (1,2,3,4,5,6)
res = filter(lambda x: x%2==0, nums)
print(res)
print(*res)

filter is a generator function. you can convert it to a list, or iterate the result from it