Sets
Basic concept. membership relation $$x\in A$$
Other concepts. Function is explained in terms of membership relation as a set $f$ of ordered pairs with $$(x,y)\in f\text{ and }(x,z)\in f \Rightarrow y=z$$
Philosophy. Sets have an inner structure - they are completely determined by their elements.
Remark. An axiomatic system widely used by set theorists is ZFC. Its strength is the simplicity: there are only sets and a membership relation. On the other hand many mathematicians feel that this leads to a set concept that diverges from their understanding and usage of sets (compare below Leinster). In fact tha vast majority of mathematicians (except set theorists) seems not to use the ZFC axioms. However, sets do not necessarily refer to ZFC (see below categories and ETCS).
Categories
Basic concept. function (arrow, morphism) $$A\rightarrow B$$
Other concepts. Membership relation $x\in A$ is explained in terms of an arrow from a terminal object (in Set a terminal object is a singleton set $\{y\})$
$$x:1\rightarrow A$$
Philosophy. Objects of a category have a priori no inner structure. They are just characterized by their relations (morphisms) to other objects.
Remark. The basic concept of categories is function and this coincides with the usage of sets by the vast majority of mathematicians. Therefore you might see categories as a conceptual generalization of the way that (most) mathematicians from very different fields use sets in their daily work. Apart from categories (and toposes) as a generalization you might have a look at the axiomatic system ETCS that is axiomatizing sets (compare below Leinster and Lawvere).
Question. What is the difference between saying x is a group, versus saying that x is in the Category Grp?
One might say the difference is not in the object $x$ itself but rather how you intend to deal with $x$.
(1) Do you ask for the inner structure of $x$? In this case it might appear natural to regard $x$ as a set.
(2) Do you ask how $x$ relates to other objects of same kind (via morphisms) or of other kind (via functors)? In this case you might tend to see $x$ as an object in the category of groups.
Critics
In case of ZFC and ETCS these approaches can be translated into each other, although ETCS is weaker than ZFC but (seemingly) covers most mathematics (see MathStackExchange and Leinster). In principle (using an extension of ETCS) you can prove the same results with both approaches. So the above mentioned philosophies of both concepts are not claiming a fundamental distinction in what you can express or what results you can prove.
The expressions set and membership in ZFC are abstract concepts just like the concepts of categories or any other axiomatic system and can mean anything. So from this formal viewpoint, to claim, that ZFC is concerned with the inner structure of sets whereas categories deal with the outer relations of objects to each other seems inappropriate. On the other hand this seems to be the philosophy or intuition of the regarding theories.
However in practice you will prefer a certain Approach for e.g. the sake of clarity or simplicity or because some concept or a connection to another area evolves more naturally than elsewhere.
References
Spivak.Category theory for scientists
Leinster.Rethinking set theory
Lawvere.An elementary theory of the category of sets
MathStackExchange.Category theory without sets