[source-klaviyo] Excessive disk use due to request cache

[Steffen911]

Connector Name

source-klaviyo

Connector Version

2.7.2

What step the error happened?

During the sync

Relevant information

When running the connector for the events table we see it filling up an events.sqlite file in the /tmp directory of the source container. After running for approximately 3 days the file grew to a total of 41GB. This causes DiskPressure on our infrastructure nodes.

This seems to be the requests-cache library with no eviction setting, i.e. all records ever received are stored on disk.

We would like to have a way to evict the cache at runtime or decide to not use a cache at all as part of the connector configuration.

I tried to delete the events.sqlite file, delete from the table, and clear the cache using the requests-cache library in a python script. In all cases, the import failed with an error due to issues with the database.

Relevant log output

No response

Contribute

• Yes, I want to contribute

[marcosmarxm]

Thank you for reporting the issue, @Steffen911. I have added it to the connector backlog for future resolution.