Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Job] Ray job log streaming misses to report the last log line #46413

Open
n30111 opened this issue Jul 3, 2024 · 0 comments
Open

[Job] Ray job log streaming misses to report the last log line #46413

n30111 opened this issue Jul 3, 2024 · 0 comments
Labels
bug Something that is supposed to be working; but isn't job triage Needs triage (eg: priority, bug/not-bug, and owning component)

Comments

@n30111
Copy link
Contributor

n30111 commented Jul 3, 2024

What happened + What you expected to happen

When streaming log from Ray Job, it fails to print the last log statement. The issue seems to be introduced in Ray 2.12 and continue to persist in 2.31. It works fine in Ray 2.10.

In the following example, it never prints the line I'm Ray

Versions / Dependencies

Ray 2.31
Python 3.11
Linux

Reproduction script

# hello.py
import os,time
print('hello')
time.sleep(2)
print("world")
time.sleep(2)
print("I'm Ray")
# submit.py
from ray.job_submission import JobSubmissionClient
client = JobSubmissionClient("http://127.0.0.1:8265")
job_id = client.submit_job(
    # Entrypoint shell command to execute
    entrypoint="python hello.py",
    # Path to the local directory that contains the script.py file
    runtime_env={"working_dir": "./"}
)
print(job_id)

logs_f = client.tail_job_logs(job_id)

async def test():
    async for line in logs_f:
        print(line)

import asyncio

asyncio.run(test())
ray start --head
python submit.py

Issue Severity

Medium: It is a significant difficulty but I can work around it.

@n30111 n30111 added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Jul 3, 2024
@anyscalesam anyscalesam added the job label Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that is supposed to be working; but isn't job triage Needs triage (eg: priority, bug/not-bug, and owning component)
Projects
None yet
Development

No branches or pull requests

2 participants