1
$\begingroup$

I was reading the documentation for Julia's Dataframe package, and the package implements Dataframe by a series of a columns each of a single type. From reading the docs, it looks like that's how R implements Dataframes as well. Is there any reason this is a better way than having a Dataframe being a collection of tuples with elements of different types? The later seems more natural to me.

What are the trade-offs or reasons one would implement Dataframes as a list of columns vs a list of rows?

$\endgroup$
1
$\begingroup$

At the level of an actual implementation, Array of structures versus structure of arrays makes an enormous performance difference; particularly when SIMD parallelism is involved. If you want to add together all of the values of one field, then all of those values are sitting next to each other.

SIMD instructions add/mul entire arrays all at once, rather than looping over each struct member. If you need to jump around memory to add all of these, you will invalidate memory caches. Also, suppose that only one column has non-default values, and the other dozen columns have zero values. The implementation can simply not bring over columns that are all zero. Run-length coding can be used on columns, or differential encoding.

In the BigData world, there are different serialization standards in use. Parquet is one option, picked because it has these characteristics due to being a columnar format. These kinds of formats are oriented towards doing stream processing. For example: you can get the next 1024 rows as a batch out of a cursor, but as an implementation detail they are packed in a columnar format.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.