issue_comments

That's so useful @sethvincent! Really interesting reading your code there, especially clever how you're using the base_url config.

I'd be very interested to see what your demo looks like without using serverless - completely agree that the less additional dependencies there are for this the better.

I'm also very interested in figuring out a way to run Datasette in Lambda but with the SQLite database on an EFS volume. Do you have a feel for how hard that would be?

Yeah the Lambda Docker stuff is pretty odd - you still don't get to speak HTTP, you have to speak their custom event protocol instead.

https://github.com/glassechidna/serverlessish looks interesting here - it adds a proxy inside the container which allows your existing HTTP Docker image to run within Docker-on-Lambda. I've not tried it out yet though.

As for your other questions:

1. I assume the goal is to have a CORS-friendly HTTPS endpoint that hosts the datasette service + user's db.

Yes, exactly. I know this will limit the size of database that can be deployed (since Lambda has a 50MB total package limit as far as I can tell) but there are plenty of interesting databases that are small enough to fit there.

The new EFS support for Lambda means that theoretically the size of database is now unlimited, which is really interesting. That's what got me inspired to take a look at a proof of concept in #850.

1. If that's the goal, I think Lambda alone is insufficient. Lambda provides the compute fabric, but not the HTTP routing. You'd also need to add Application Load Balancer or API Gateway to provide an HTTP endpoint that routes to the lambda function.

Do you have a preference between ALB or API GW? ALB has better economics at scale, but has a minimum monthly cost. API GW has worse per-request economics, but scales to zero when no requests are happening.

I personally like scale-to-zero because many of my projects are likely to receive very little traffic. So API GW first, and maybe ALB as an option later on for people operating at scale?

1. Does Datasette have any native components, or is it all pure python? If it has native bits, they'll likely need to be recompiled to work on Amazon Linux 2.

As you've found, the only native component is uvloop which is only needed if uvicorn is being used to serve requests.

1. There are a few disparate services that need to be wired together to expose a Python service securely to the web. If I was doing this outside of the datasette publish system, I'd use an AWS CloudFormation template. Even within datasette, I think it still makes sense to use a CloudFormation template and just have the publish plugin invoke it (via the standard aws cli) with user-specified parameters. Does that sound reasonable to you?

For the eventual "datasette publish lambda" command I want whatever results in the smallest amount of inconvenience for users. I've been trying out Amazon SAM in #850 and it requires users to run Docker on their machines, which is a pretty huge barrier to entry! I don't have much experience with CloudFormation but it's probably a better bet, especially if you can "pip install" the dependencies needed to deploy with it.

Hi Colin,

Sorry I didn't see this sooner! I've just started digging into this myself, to try and play with the new EFS Lambda support: #850.

Yes, uvloop is only needed because of uvicorn. I have a branch here that removes that dependency just for trying out Lambda: https://github.com/simonw/datasette/tree/no-uvicorn - so you can run pip install https://github.com/simonw/datasette/archive/no-uvicorn.zip to get that.

I'm going to try out your datasette-lambda project next - really excited to see how far you've got with it.