Definitely rude taking the bin from the property.
As others have said, in our court, we have an unspoken agreement that if your bin is out for collection it's fair game - we hear bins moving about late at night as various neighbours hunt around :D
However one day we went to put our green...