I think it feels like a competition for celery. It fits into those tasks that are too heavy to run on the primary web server, but too small to set up the dedicated deployment/replicaset.
On slightly different note, I am still looking for a good job pattern for my use case. I run jobs on very custom docker images (including deep learning and computer vision stuff). I currently use redis queue with todo items, each pod in deployment picks up items from the queue in the loop, but I dislike some things about it. Fission is not the good fit, but I am wondering if HN community would have some suggestions. I thought about kubernetes jobs, but they have some things I don't like.
If you can model your data flow as pipelines that run when their inputs update, you might interested in Pachyderm (https://www.pachyderm.io/). This uses Kubernetes and Docker images to run multi-stage computations. It's still a bit rough around the edges. (I wouldn't recommend the current version if you work with multi-gigabyte binary input files, though they're currently working on fixing an issue affecting that use case.) They do have some examples (http://pachyderm.readthedocs.io/en/latest/examples/readme.ht...), including one using TensorFlow.
We're experimenting with Pachyderm at work, and the design feels "right" for our use case.
- As mentioned previously, retry count limit to job/pod would be useful.
- Minor nit: Deleting old job with the pod is cumbersome with current python API. Either delete_namespaced_job with orphan_dependencies: False, but I would need to check if job exists beforehand (deleting if job does not exist fails). On the other hand delete_collection_namespaced_job lets me delete job by label selector, but I also need to manually clean up the pod.
- Another nit: When exporting stackdriver logs collected by GKE from a job to big query, each log ends up as a separate logsource.
- Retry control is course grained. IIUC the only option are "no retry at all" or "terminate if job have been active for X seconds". If my jobs tend to be long and failure happens early, second option can lead to hundreds of retries. Each job retry is expensive, as it creates a new pod. I remember ending up with 100s pods.
Not the OP, but I'll chime in with what we're missing when we took a look at it. AFAICT there is no way to tell k8s to retry a job X number of times; you have to futz around with deadlines.
Job ordering/dependencies! Job1a must finish successfully before Job1b and Job1c.
Alternatively, all Jobs matching this query must succeed before this job. Failures can be OK if a configurable threshold of success is met. Exceeding the thresh fails the dependent jobs before they get to run.
You can do all of this with supervisor pods, but since you asked...
Redis itself is fine, but job queue around redis is finnicky.
On one hand, celery and python-rq do not work well big custom jobs (they are more suited for small tasks, but too big to run on the web server). On the other hand, using raw redis queue is finicky if I want to implement custom things like retries, timeouts, etc.
On slightly different note, I am still looking for a good job pattern for my use case. I run jobs on very custom docker images (including deep learning and computer vision stuff). I currently use redis queue with todo items, each pod in deployment picks up items from the queue in the loop, but I dislike some things about it. Fission is not the good fit, but I am wondering if HN community would have some suggestions. I thought about kubernetes jobs, but they have some things I don't like.