FEAT-#4605: Add native query compiler #7259
Conversation
arunjose696
requested review from
devin-petersohn,
mvashishtha,
RehanSD,
YarShev,
vnlitvinov,
anmyachev,
dchigarev and
a team
as code owners
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
arunjose696
changed the title
arunjose696
force-pushed
the
arun-sqc
branch
3 times, most recently
from
f80e353
to
41bab97
Compare
modin/experimental/core/storage_formats/pandas/small_query_compiler.py
Outdated
Show resolved
Hide resolved
modin/config/envvars.py
Outdated
| @@ -851,4 +851,11 @@ def _check_vars() -> None: | |||
| ) | |||
|
|
|||
|
|
|||
| class UsePlainPandasQueryCompiler(EnvironmentVariable, type=bool): | |||
There was a problem hiding this comment.
This name is probably a little confusing for users. I suggest something like SmallDataframeMode. This can be set to None by default, and users can set it to "pandas" or some other option in the future (we may have some other single node options coming).
There was a problem hiding this comment.
@devin-petersohn, do you think VanillaPandasMode is a good option? Also, why do you think we should make this config of string type to have choices None/pandas/etc.? Wouldn't it be sufficient to have this config boolean - enable/disable?
There was a problem hiding this comment.
In the future we may add polars mode. If this happens, we might also want to have an option for that. Making it a string keeps it open to other options. If we have pandas in the name, we can only use that mode for pandas execution. I'm open to other names, but I think we don't want to keep adding more and more configs if we have more options later.
There was a problem hiding this comment.
Doesn't this sound like we may have multiple storage formats for a single execution? Do we really want to support this in future?
There was a problem hiding this comment.
Potentially, yes I think this is something we could support in the future.
There was a problem hiding this comment.
@devin-petersohn, do you think we could support automatic initialization with small qc depending on a data size threshold in future?
There was a problem hiding this comment.
I propose to rename UsePlainPandasQueryCompiler to NativeDataframeMode and SmallQueryCompiler to NativeQueryCompiler by sort of analogy with HdkOnNative we had previously.
There was a problem hiding this comment.
At a minimum, a more complete definition of this class in the docstring is required.
There was a problem hiding this comment.
I will update the name to UsePlainPandasQueryCompiler to NativeDataframeMode and SmallQueryCompiler to NativeQueryCompiler.
The most changes in tests are disabling few checks as it wont be supported without partitions, and as the current changes dont yet support IO like pd.read_csv(), Is there something specific that should be avoided? |
Nothing specific, I was just trying to understand context. Thanks! |
arunjose696
force-pushed
the
arun-sqc
branch
2 times, most recently
from
e6b035f
to
d406414
Compare
…rialized_dtypes to query compiler layer as in the code in multiple places the methods of private _modin_frame were used
Signed-off-by: Igoshev, Iaroslav <[email protected]>
Co-authored-by: Iaroslav Igoshev <[email protected]> Signed-off-by: arunjose696 <[email protected]>
Co-authored-by: Iaroslav Igoshev <[email protected]>
There was a problem hiding this comment.
@arunjose696, please fix ci-required / lint (pydocstyle) (pull_request) job. Other than that, LGTM!
arunjose696
force-pushed
the
arun-sqc
branch
4 times, most recently
from
3da8c6e
to
17b12aa
Compare
Signed-off-by: arunjose696 <[email protected]>
YarShev
changed the title
What do these changes do?
flake8 modin/ asv_bench/benchmarks scripts/doc_checker.pyblack --check modin/ asv_bench/benchmarks scripts/doc_checker.pygit commit -sdocs/development/architecture.rstis up-to-date