Skip to content

feat: allow functions decorated with @bpd.remote_function to execute locally#704

Merged
tswast merged 7 commits intomainfrom
tswast-remote-function-local-testing
May 26, 2024
Merged

feat: allow functions decorated with @bpd.remote_function to execute locally#704
tswast merged 7 commits intomainfrom
tswast-remote-function-local-testing

Conversation

@tswast
Copy link
Contributor

@tswast tswast commented May 17, 2024 •
edited
Loading

BEGIN_COMMIT_OVERRIDE
feat: allow functions decorated with bpd.remote_function() to execute locally (#704)
END_COMMIT_OVERRIDE

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

🦕

@tswast tswast requested a review from a team as a code owner May 17, 2024 21:09
@tswast tswast requested review from a team and ashleyxuu May 17, 2024 21:09
@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. labels May 17, 2024
@tswast tswast mentioned this pull request May 17, 2024
Merged
5 tasks
@tswast
Copy link
Contributor Author

tswast commented May 20, 2024

e2e failure tests/system/large/test_remote_function.py::test_remote_function_stringify_with_ibis is a real one. We don't have the functions return ibis objects anymore, so that test will need to be updated.

I'm not sure what's going on with the Could not create or update Cloud Run service bigframes-9c95c91485a237f4d58dba298ce3b103, Container Healthcheck failed. failures. Will investigate further.

@tswast
Copy link
Contributor Author

tswast commented May 20, 2024

Regarding the other failures, looks like the functions were failing to start due to:

ERROR 2024-05-17T22:49:25.369026Z Traceback (most recent call last): File "/layers/google.python.pip/pip/bin/functions-framework", line 8, in <module> sys.exit(_cli())
DEFAULT 2024-05-17T22:49:25.369047Z ^^^^^^
DEFAULT 2024-05-17T22:49:25.369055Z File "/layers/google.python.pip/pip/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
DEFAULT 2024-05-17T22:49:25.370216Z return self.main(*args, **kwargs)
DEFAULT 2024-05-17T22:49:25.370455Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-05-17T22:49:25.370469Z File "/layers/google.python.pip/pip/lib/python3.12/site-packages/click/core.py", line 1078, in main
DEFAULT 2024-05-17T22:49:25.370797Z rv = self.invoke(ctx)
DEFAULT 2024-05-17T22:49:25.370941Z ^^^^^^^^^^^^^^^^
DEFAULT 2024-05-17T22:49:25.370949Z File "/layers/google.python.pip/pip/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
DEFAULT 2024-05-17T22:49:25.371290Z return ctx.invoke(self.callback, **ctx.params)
DEFAULT 2024-05-17T22:49:25.371504Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-05-17T22:49:25.371513Z File "/layers/google.python.pip/pip/lib/python3.12/site-packages/click/core.py", line 783, in invoke
DEFAULT 2024-05-17T22:49:25.371739Z return __callback(*args, **kwargs)
DEFAULT 2024-05-17T22:49:25.372052Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-05-17T22:49:25.372065Z File "/layers/google.python.pip/pip/lib/python3.12/site-packages/functions_framework/_cli.py", line 37, in _cli
DEFAULT 2024-05-17T22:49:25.374076Z app = create_app(target, source, signature_type)
DEFAULT 2024-05-17T22:49:25.374381Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-05-17T22:49:25.374391Z File "/layers/google.python.pip/pip/lib/python3.12/site-packages/functions_framework/__init__.py", line 288, in create_app
DEFAULT 2024-05-17T22:49:25.375787Z spec.loader.exec_module(source_module)
DEFAULT 2024-05-17T22:49:25.375854Z File "<frozen importlib._bootstrap_external>", line 995, in exec_module
DEFAULT 2024-05-17T22:49:25.375865Z File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
DEFAULT 2024-05-17T22:49:25.375872Z File "/workspace/main.py", line 9, in <module>
DEFAULT 2024-05-17T22:49:25.375963Z udf = cloudpickle.load(f)
DEFAULT 2024-05-17T22:49:25.376108Z ^^^^^^^^^^^^^^^^^^^
WARNING 2024-05-17T22:49:26.383786824Z Container called exit(1).
DEFAULT 2024-05-17T22:49:26.383934Z ModuleNotFoundError: No module named 'pandas'

I wonder if it's because we're pickling more than we used to because the function body is now intact? Will look a bit deeper.

@tswast
Copy link
Contributor Author

tswast commented May 20, 2024

I wonder if it's because we're pickling more than we used to because the function body is now intact? Will look a bit deeper.

Looks like this only affected some specific tests that basically decorated the same function twice. Should be fixed by cc11de5

@tswast
Copy link
Contributor Author

tswast commented May 20, 2024

Remaining failures:

FAILED tests/system/large/test_remote_function.py::test_remote_function_via_session_vpc_invalid
FAILED tests/system/large/ml/test_ensemble.py::test_xgbclassifier_dart_booster_multiple_params
FAILED tests/system/large/ml/test_ensemble.py::test_randomforestclassifier_multiple_params

I don't think test_ensemble.py tests are related, but I will need to investigate all of these more.

ashleyxuu
ashleyxuu reviewed May 20, 2024
View reviewed changes
Copy link
Contributor

@ashleyxuu ashleyxuu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_remote_function_via_session_vpc_invalid Is this related to your change?

@tswast
Copy link
Contributor Author

tswast commented May 20, 2024

Re: test_remote_function_via_session_vpc_invalid

This is also failing in other PRs unrelated to this change. Filed internal issue 341772893 to investigate.

@tswast tswast requested review from GarrettWu and ashleyxuu and removed request for ashleyxuu May 24, 2024 14:45
@tswast tswast enabled auto-merge (squash) May 25, 2024 01:45
@tswast tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label May 26, 2024
@bigframes-bot bigframes-bot removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label May 26, 2024
@tswast tswast merged commit d850da6 into main May 26, 2024
@tswast tswast deleted the tswast-remote-function-local-testing branch May 26, 2024 20:10
@release-please release-please bot mentioned this pull request May 26, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@GarrettWu GarrettWu GarrettWu approved these changes

+1 more reviewer

@ashleyxuu ashleyxuu ashleyxuu left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@shobsi shobsi

Labels

api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. size: m Pull request size is medium.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@tswast @GarrettWu @ashleyxuu @shobsi @bigframes-bot