[Core] Optionally allow user code to directly produce data into a plasma store managed buffer

[Superskyyy]

Description

Since Plasma was deprecated from Arrow and Ray uses a forked version that is heavily modified, I'm not sure there's an existing way to directly produce data into a fix-sized buffer that lives inside of the object store. Doing so, although is an advanced pattern, would allow heavy data loading operations to save one unnecessary memcpy from the heap and therefore potentially save a lot of CPU time.

Use case

One important use case is to support rapid saving of large model checkpoints directly into the Plasma store without having to save it to the heap first. Then an guardian checkpointer actor (i.e., take care of checkpoint lifecycle and support fault tolerance) can take its time transferring the checkpoint elsewhere, into distributed storage (other plasma stores, disk or network drive), which forms a reliable layered checkpointing system. Whenever trains fail, training process can either recover from 1. plasma 2. local disk 3. remote plasma or network drives. depending on the failure level.

Another use case is when an external datasource might support some kind of (direct memory access) DMA operation, which should be able to directly persist into plasma instead of costing an additional memcpy.

These are just my initial thoughts, any comments and feedback are welcome. @Bye-legumes @liuxsh9 @nemo9cby