I'm working on a project where my application has to ingest data from a huge number of different data sources. Each of these data sources has their own mechanism for auth (usually OAuth2, but not always), some send metadata and require a callback, some have "resources updated_since" type endpoints that need to be polled periodically, some required waiting on a data export to finish, and then collecting that data. There are a bunch of different ways in which we might be sent data, and there are many more coming.

Note: The data sources themselves are partners, so it's possible to push some of the work onto their side, as long as the mechanism for having them send data doesn't seriously affect the way their systems work.

Possible solutions

Build an integration for each data source

This seems overkill, and would introduce a lot of redundancy. Trying to avoid this.

Build a configuration abstraction API(s)/UI

Token management

For example, I could imagine a simple endpoint for configuring auth for a data source. Eg: here's the token refresh endpoint, here's the client_id and secret, here's the token lifespan etc. And then our system would just manage all those tokens automatically.

Data source config

Things get more complex on the actual data source side. It seems like creating a configuration system with a bunch of different modular steps might make sense, eg: step for receiving a list of IDs, step for querying a specific ID, step for polling a data generation status endpoint, step for collecting updated_since data on a particular interval etc.

Final Thoughts

Are you aware of any companies/systems similar to this? Are there certain design patterns that make sense here? Any gotchas to be aware of?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.