• [x] passes make lint
• [x] adds tests to tests/ (if appropriate)

What does this PR change?

Automatically add saturn-specific labels to the flow environment.

How does this PR improve prefect-saturn?

Along with the related change in saturn, this allows flows to be assigned to the correct cluster agent even if there are agents in multiple clusters that are all using the same prefect-cloud tenant.

• [X] passes make lint
• [X] adds tests to tests/ (if appropriate)

What does this PR change?

• Bumps version to 0.5.1.9000 for development

How does this PR improve prefect-saturn?

With this change, users can rely on the behavior that installations from source control will have a newer version than the newest release available on PyPI, but guaranteed to be older than the next release on PyPI.

• [x] passes make lint
• [x] adds tests to tests/ (if appropriate)

What does this PR change?

Adds instance size argument for flow node

How does this PR improve prefect-saturn?

Enables users to select a larger node size for their flow to run on. This means we are not forcing users to farm all of the work out to dask-clusters, they can instead run flows with a local executor and not be constrained by a medium tier node.

Without this PR, modifying the python version could lead to different hashes for flow metadata. This PR hashes identifying information for flows using json, rather than pickle. This PR does change the hashing function. As a result re-registering a flow will create a new flow in Saturn Cloud.

• [x] passes make lint
• [x] adds tests to tests/ (if appropriate)

What does this PR change?

Changes PrefectCloudIntegration to expect the environment BASE_URL to NOT end in a trailing slash. In all Saturn production environments, BASE_URL does not end in a trailing slash.

This PR also bumps the version of prefect-saturn to 0.2.0, since this change in behavior is a breaking change.

How does this PR improve prefect-saturn?

Without this fix, prefect-saturn is currently broken in Saturn production environments.

Why can't we just keep the code that adds or removes slashes as needed?

This project uses prefect's WebhookStorage (https://docs.prefect.io/orchestration/execution/storage_options.html#webhook). That type of storage uses template strings to reference environment variables, like this:

flow = Flow(
    "some-flow",
    storage=Webhook(
        build_request_kwargs={
            "url": "${BASE_URL}/api/whatever",
        },
        build_request_http_method="POST",
       ....
    )
)

There is not place in there where we can introduce Saturn-written code that checks the environment variable BASE_URL and deals with missing or extra trailing slashes. So an assumption about whether or not the environment variable BASE_URL ends in a slash has to be hard-coded into the codebase here.

• [x] passes make lint
• [ ] adds tests to tests/ (if appropriate)

I don't believe any tests need to be added.

What does this PR change?

This PR changes the Client()._active_tenant_id > Client().tenant_id (when using Prefect >= 0.15) since _active_tenant_id is no longer a property of Prefect's Client in 0.15.

How does this PR improve prefect-saturn?

This allows prefect-saturn to work with prefect in version 0.15 which removed the private attribute.

Unlike PyYAML, ruamel.yaml supports:

• YAML <= 1.2. PyYAML only supports YAML <= 1.1 This is vital, as YAML 1.2 intentionally breaks backward compatibility with YAML 1.1 in several edge cases. This would usually be a bad thing. In this case, this renders YAML 1.2 a strict superset of JSON. Since YAML 1.1 is not a strict superset of JSON, this is a good thing.
• Roundtrip preservation When calling yaml.dump() to dump a dictionary loaded by a prior call to yaml.load():

See more details at https://yaml.readthedocs.io/en/latest/pyyaml.html

• [x] passes make lint
• [x] adds tests to tests/ (if appropriate)

What does this PR change?

This PR replaces Docker storage with Webhook storage.

This PR also adds a small working example to the README, to show users how it works. A longer-form tutorial will be up on https://www.saturncloud.io/docs/ some time in the next week.

How does this PR improve prefect-saturn?

The use of Webhook storage will make the integration between Prefect Cloud and Saturn Cloud much faster and less error-prone. It eliminates several hacks and special cases, should be much quicker, and removes an awkward race condition. With Docker storage, after calling .register_flow_with_saturn() you had to wait for a k8s job that built and pushed the image to complete. That could take up to 10 minutes, and nothing in prefect-saturn or the Saturn UI allowed you to see logs or other progress of that job.

That job was also very brittle...it required docker-in-docker trickery, patching user-chosen images with several build-time-only dependencies, and running a sequence of multiple gnarly shell scripts.

With Webhook storage, all of that complexity is eliminated. When you call .register_flow_with_saturn(), the flow is serialized with cloudpickle, sent to Saturn, and stored as bytes in an object store. At run time, when flow.storage.get_flow() is called, Saturn retrieves the binary content of the flow from an object store and sends it back over the wire. That's it! Everything is synchronous, so no weird race conditions, and it's just passing bytes around, so the storage process goes from 10+ minutes to under a second.

• [x] passes make lint
• [x] adds tests to tests/ (if appropriate)

What does this PR change?

• removes fields from saturn_details that are unnecessary as of #8
• renames saturn_details to storage_details since that now only contains information for building storage
• bump version floor to Python 3.7 (all the Saturn images are Python 3.7)
• set ignore_healthchecks=True on the storage object.
  • This avoids a weird error where the image being built with flow.storage.build() doesn't have stuff like the Saturn start script in it, which could cause errors.
  • We don't need to care about that because every job in the flow's lifecycle will have the start script added to it's command / args

How does this PR improve prefect-saturn?

This PR removes unnecessary fields that could have become dependencies in users' code, reducing the surface for breaking changes.

The ignore_healthchecks thing makes it possible for users to rely on the start script to install libraries.

• [x] passes make lint
• [x] adds tests to tests/ (if appropriate)

What does this PR change?

This PR changes the strategy for uniquely identifying flows. Now the flow_hash sent to Saturn Cloud is the sha256 hash of:

• project name
• flow name
• Prefect Cloud tenant id

This means that flow_hash is equivalent to the Prefect Cloud concept flow_group_id. This PR proposes creating this hash ourselves because Prefect Cloud's flow_group_id can't be known until you've registered a flow with Prefect Cloud, and prefect-saturn needs to register with Saturn first.

How does this PR improve prefect-saturn?

This PR provides a reliable way to uniquely identify all versions of the same flow. It improves on the previous model, where tenant id was not considered. The previous model could have caused conflicts in the case where two flows with identical names and code, in Prefect Cloud projects with the same name but in different tenants, would get the same hash and conflict.

Because this PR no longer considers the task graph in the hash, it also means that the hash will not change as a flow's task graph changes. That means pushing Docker storage to a container registry should be a lot faster, since it'll be more likely to hit the registry's cache.

Notes for Reviewers

I had to introduce prefect.client.Client in this PR, and then mock it with patch(). A lot of the diff in the test files is just whitespace, the result of adding in a with patch(....). I recommend reviewing with whitespace changes hidden.

This PR includes a few updates that improve execution and testing for the end-to-end experience with Prefect Cloud.

What does this PR change?

This PR adds more details from Saturn to the flow, so the agent running it knows how to configure the first job that loads the flow.

How does this PR improve prefect-saturn?

This PR allows flows executed from Prefect Cloud to take advantage of Saturn-y customization features like env secrets, filesystem secrets, and a custom start script.