issue_comments

25 rows where issue = 639993467 sorted by updated_at descending

View and edit SQL

Suggested facets: reactions, created_at (date), updated_at (date)

user

issue

  • Proof of concept for Datasette on AWS Lambda with EFS · 25

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
645068128 https://github.com/simonw/datasette/issues/850#issuecomment-645068128 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA2ODEyOA== simonw 9599 2020-06-16T23:52:16Z 2020-06-16T23:52:16Z OWNER

https://aws.amazon.com/blogs/compute/announcing-http-apis-for-amazon-api-gateway/ looks very important here: AWS HTTP APIs were introduced in December 2019 and appear to be a third of the price of API Gateway.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645064332 https://github.com/simonw/datasette/issues/850#issuecomment-645064332 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA2NDMzMg== simonw 9599 2020-06-16T23:37:34Z 2020-06-16T23:37:34Z OWNER

Just realized Colin Dellow reported an issue with Datasette and Mangum back in April - #719 - and has in fact been working on https://github.com/code402/datasette-lambda for a while!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645063386 https://github.com/simonw/datasette/issues/850#issuecomment-645063386 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA2MzM4Ng== simonw 9599 2020-06-16T23:34:07Z 2020-06-16T23:34:07Z OWNER

Tried sam local invoke:

simon@Simons-MacBook-Pro datasette-proof-of-concept % sam local invoke
Invoking app.lambda_handler (python3.8)

Fetching lambci/lambda:python3.8 Docker container image......
Mounting /private/tmp/datasette-proof-of-concept/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: 7c04480b-5d42-168e-dec0-4e8bf34fa596 Version: $LATEST
[INFO]  2020-06-16T23:33:27.24Z 7c04480b-5d42-168e-dec0-4e8bf34fa596    Waiting for application startup.
[INFO]  2020-06-16T23:33:27.24Z 7c04480b-5d42-168e-dec0-4e8bf34fa596    LifespanCycleState.STARTUP:  'lifespan.startup.complete' event received from application.
[INFO]  2020-06-16T23:33:27.24Z 7c04480b-5d42-168e-dec0-4e8bf34fa596    Application startup complete.
[INFO]  2020-06-16T23:33:27.24Z 7c04480b-5d42-168e-dec0-4e8bf34fa596    Waiting for application shutdown.
[INFO]  2020-06-16T23:33:27.24Z 7c04480b-5d42-168e-dec0-4e8bf34fa596    LifespanCycleState.SHUTDOWN:  'lifespan.shutdown.complete' event received from application.
[ERROR] KeyError: 'requestContext'
Traceback (most recent call last):
  File "/var/task/mangum/adapter.py", line 110, in __call__
    return self.handler(event, context)
  File "/var/task/mangum/adapter.py", line 130, in handler
    if "eventType" in event["requestContext"]:
END RequestId: 7c04480b-5d42-168e-dec0-4e8bf34fa596
REPORT RequestId: 7c04480b-5d42-168e-dec0-4e8bf34fa596  Init Duration: 1120.76 ms   Duration: 7.08 ms   Billed Duration: 100 ms Memory Size: 128 MBMax Memory Used: 47 MB   

{"errorType":"KeyError","errorMessage":"'requestContext'","stackTrace":["  File \"/var/task/mangum/adapter.py\", line 110, in __call__\n    return self.handler(event, context)\n","  File \"/var/task/mangum/adapter.py\", line 130, in handler\n    if \"eventType\" in event[\"requestContext\"]:\n"]}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645062266 https://github.com/simonw/datasette/issues/850#issuecomment-645062266 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA2MjI2Ng== simonw 9599 2020-06-16T23:30:12Z 2020-06-16T23:33:12Z OWNER

OK, changed requirements.txt to this:

https://github.com/simonw/datasette/archive/no-uvicorn.zip
mangum

No sam build --use-container runs without errors. Ran sam deploy too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645063058 https://github.com/simonw/datasette/issues/850#issuecomment-645063058 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA2MzA1OA== simonw 9599 2020-06-16T23:32:57Z 2020-06-16T23:32:57Z OWNER

https://q7lymja3sj.execute-api.us-east-1.amazonaws.com/Prod/hello/ is now giving me a 500 internal server error.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645061088 https://github.com/simonw/datasette/issues/850#issuecomment-645061088 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA2MTA4OA== simonw 9599 2020-06-16T23:25:41Z 2020-06-16T23:25:41Z OWNER

Someone else ran into this problem: https://github.com/iwpnd/fastapi-aws-lambda-example/issues/1

So I need to be able to pip install MOST of Datasette, but skip uvicorn. Tricky. I'll try installing a custom fork?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645060598 https://github.com/simonw/datasette/issues/850#issuecomment-645060598 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA2MDU5OA== simonw 9599 2020-06-16T23:24:01Z 2020-06-16T23:24:01Z OWNER

I changed requirements.txt to this:

datasette
mangum

And app.py to this:

from datasette.app import Datasette
from mangum import Mangum


datasette = Datasette([], memory=True)
lambda_handler = Mangum(datasette.app())

But then when I ran sam build --use-container I got this:

simon@Simons-MacBook-Pro datasette-proof-of-concept % sam build --use-container
Starting Build inside a container
Building function 'HelloWorldFunction'

Fetching lambci/lambda:build-python3.8 Docker container image......
Mounting /private/tmp/datasette-proof-of-concept/hello_world as /tmp/samcli/source:ro,delegated inside runtime container

Build Failed
Running PythonPipBuilder:ResolveDependencies
Error: PythonPipBuilder:ResolveDependencies - {uvloop==0.14.0(wheel)}

uvloop isn't actually necessary for this project, since it's used by uvicorn which isn't needed if Lambda is serving ASGI traffic directly.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645059663 https://github.com/simonw/datasette/issues/850#issuecomment-645059663 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA1OTY2Mw== simonw 9599 2020-06-16T23:20:46Z 2020-06-16T23:20:46Z OWNER

I added an exclamation mark to hello world and ran sam deploy again. https://q7lymja3sj.execute-api.us-east-1.amazonaws.com/Prod/hello/ still shows the old message.

Running sam build --use-container first and then sam deploy did the right thing.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645058947 https://github.com/simonw/datasette/issues/850#issuecomment-645058947 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA1ODk0Nw== simonw 9599 2020-06-16T23:18:18Z 2020-06-16T23:18:18Z OWNER

https://q7lymja3sj.execute-api.us-east-1.amazonaws.com/Prod/hello/

That's a pretty ugly URL. I'm not sure how to get rid of the /Prod/ prefix on it. Might have to use the base_url setting to get something working: https://datasette.readthedocs.io/en/stable/config.html#base-url

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645058617 https://github.com/simonw/datasette/issues/850#issuecomment-645058617 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA1ODYxNw== simonw 9599 2020-06-16T23:17:09Z 2020-06-16T23:17:09Z OWNER

OK, sam deploy --guided now works!

simon@Simons-MacBook-Pro datasette-proof-of-concept % sam deploy --guided

Configuring SAM deploy
======================

    Looking for samconfig.toml :  Not found

    Setting default arguments for 'sam deploy'
    =========================================
    Stack Name [sam-app]: datasette-proof-of-concept
    AWS Region [us-east-1]: 
    #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
    Confirm changes before deploy [y/N]: 
    #SAM needs permission to be able to create roles to connect to the resources in your template
    Allow SAM CLI IAM role creation [Y/n]: 
    HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
    Save arguments to samconfig.toml [Y/n]: 

    Looking for resources needed for deployment: Not found.
    Creating the required resources...
    Successfully created!

        Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-1ksajo4h62s07
        A different default S3 bucket can be set in samconfig.toml

    Saved arguments to config file
    Running 'sam deploy' for future deployments will use the parameters saved above.
    The above parameters can be changed by modifying samconfig.toml
    Learn more about samconfig.toml syntax at 
    https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

    Deploying with following values
    ===============================
    Stack name                 : datasette-proof-of-concept
    Region                     : us-east-1
    Confirm changeset          : False
    Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-1ksajo4h62s07
    Capabilities               : ["CAPABILITY_IAM"]
    Parameter overrides        : {}

Initiating deployment
=====================
Uploading to datasette-proof-of-concept/0c208b5656a7aeb6186d49bebc595237  535344 / 535344.0  (100.00%)
HelloWorldFunction may not have authorization defined.
Uploading to datasette-proof-of-concept/14bd9ce3e21f9c88634d13c0c9b377e4.template  1147 / 1147.0  (100.00%)

Waiting for changeset to be created..

CloudFormation stack changeset
---------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                           LogicalResourceId                                   ResourceType                                      
---------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add                                               HelloWorldFunctionHelloWorldPermissionProd          AWS::Lambda::Permission                           
+ Add                                               HelloWorldFunctionRole                              AWS::IAM::Role                                    
+ Add                                               HelloWorldFunction                                  AWS::Lambda::Function                             
+ Add                                               ServerlessRestApiDeployment47fc2d5f9d               AWS::ApiGateway::Deployment                       
+ Add                                               ServerlessRestApiProdStage                          AWS::ApiGateway::Stage                            
+ Add                                               ServerlessRestApi                                   AWS::ApiGateway::RestApi                          
---------------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:us-east-1:462092780466:changeSet/samcli-deploy1592349262/d685f2de-87c1-4b8e-b13a-67b94f8fc928


2020-06-16 16:14:29 - Waiting for stack create/update to complete

CloudFormation events from changeset
---------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                         ResourceType                           LogicalResourceId                      ResourceStatusReason                 
---------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS                     AWS::IAM::Role                         HelloWorldFunctionRole                 -                                    
CREATE_IN_PROGRESS                     AWS::IAM::Role                         HelloWorldFunctionRole                 Resource creation Initiated          
CREATE_COMPLETE                        AWS::IAM::Role                         HelloWorldFunctionRole                 -                                    
CREATE_IN_PROGRESS                     AWS::Lambda::Function                  HelloWorldFunction                     Resource creation Initiated          
CREATE_IN_PROGRESS                     AWS::Lambda::Function                  HelloWorldFunction                     -                                    
CREATE_COMPLETE                        AWS::Lambda::Function                  HelloWorldFunction                     -                                    
CREATE_IN_PROGRESS                     AWS::ApiGateway::RestApi               ServerlessRestApi                      Resource creation Initiated          
CREATE_IN_PROGRESS                     AWS::ApiGateway::RestApi               ServerlessRestApi                      -                                    
CREATE_COMPLETE                        AWS::ApiGateway::RestApi               ServerlessRestApi                      -                                    
CREATE_IN_PROGRESS                     AWS::Lambda::Permission                HelloWorldFunctionHelloWorldPermissi   -                                    
                                                                              onProd                                                                      
CREATE_IN_PROGRESS                     AWS::ApiGateway::Deployment            ServerlessRestApiDeployment47fc2d5f9   -                                    
                                                                              d                                                                           
CREATE_COMPLETE                        AWS::ApiGateway::Deployment            ServerlessRestApiDeployment47fc2d5f9   -                                    
                                                                              d                                                                           
CREATE_IN_PROGRESS                     AWS::ApiGateway::Deployment            ServerlessRestApiDeployment47fc2d5f9   Resource creation Initiated          
                                                                              d                                                                           
CREATE_IN_PROGRESS                     AWS::Lambda::Permission                HelloWorldFunctionHelloWorldPermissi   Resource creation Initiated          
                                                                              onProd                                                                      
CREATE_IN_PROGRESS                     AWS::ApiGateway::Stage                 ServerlessRestApiProdStage             -                                    
CREATE_COMPLETE                        AWS::ApiGateway::Stage                 ServerlessRestApiProdStage             -                                    
CREATE_IN_PROGRESS                     AWS::ApiGateway::Stage                 ServerlessRestApiProdStage             Resource creation Initiated          
CREATE_COMPLETE                        AWS::Lambda::Permission                HelloWorldFunctionHelloWorldPermissi   -                                    
                                                                              onProd                                                                      
CREATE_COMPLETE                        AWS::CloudFormation::Stack             datasette-proof-of-concept             -                                    
---------------------------------------------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
---------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs                                                                                                                                                 
---------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 HelloWorldFunctionIamRole                                                                                                           
Description         Implicit IAM Role created for Hello World function                                                                                  
Value               arn:aws:iam::462092780466:role/datasette-proof-of-concept-HelloWorldFunctionRole-8MIDNIV5ECA6                                       

Key                 HelloWorldApi                                                                                                                       
Description         API Gateway endpoint URL for Prod stage for Hello World function                                                                    
Value               https://q7lymja3sj.execute-api.us-east-1.amazonaws.com/Prod/hello/                                                                  

Key                 HelloWorldFunction                                                                                                                  
Description         Hello World Lambda Function ARN                                                                                                     
Value               arn:aws:lambda:us-east-1:462092780466:function:datasette-proof-of-concept-HelloWorldFunction-QTF78ZEUDCB                            
---------------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - datasette-proof-of-concept in us-east-1
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645056636 https://github.com/simonw/datasette/issues/850#issuecomment-645056636 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA1NjYzNg== simonw 9599 2020-06-16T23:10:22Z 2020-06-16T23:10:22Z OWNER

Clicking that button generated me an access key ID / access key secret pair. Dropping those into ~/.aws/credentials using this format:

[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645055200 https://github.com/simonw/datasette/issues/850#issuecomment-645055200 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA1NTIwMA== simonw 9599 2020-06-16T23:05:48Z 2020-06-16T23:05:48Z OWNER

Logged in as simon-administrator I'm using https://console.aws.amazon.com/iam/home?region=us-east-2#/security_credentials to create credentials:

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645054206 https://github.com/simonw/datasette/issues/850#issuecomment-645054206 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA1NDIwNg== simonw 9599 2020-06-16T23:02:54Z 2020-06-16T23:04:59Z OWNER

I think I need to sign in to the AWS console with this new simon-administrator account and create IAM credentials for it.

... for which I needed my root "account ID" - a 12 digit number - to use on the IAM login form.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645053923 https://github.com/simonw/datasette/issues/850#issuecomment-645053923 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA1MzkyMw== simonw 9599 2020-06-16T23:01:49Z 2020-06-16T23:01:49Z OWNER

I used https://console.aws.amazon.com/billing/home?#/account and activated "IAM user/role access to billing information" - what a puzzling first step!

I created a new user with AWS console access (which means access to the web UI) called simon-administrator and set a password. I created an Administrators group with AdministratorAccess.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645051972 https://github.com/simonw/datasette/issues/850#issuecomment-645051972 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA1MTk3Mg== simonw 9599 2020-06-16T22:55:04Z 2020-06-16T22:55:04Z OWNER
simon@Simons-MacBook-Pro datasette-proof-of-concept % sam deploy --guided

Configuring SAM deploy
======================

    Looking for samconfig.toml :  Not found

    Setting default arguments for 'sam deploy'
    =========================================
    Stack Name [sam-app]: datasette-proof-of-concept
    AWS Region [us-east-1]: 
    #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
    Confirm changes before deploy [y/N]: y
    #SAM needs permission to be able to create roles to connect to the resources in your template
    Allow SAM CLI IAM role creation [Y/n]: y
    HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
    Save arguments to samconfig.toml [Y/n]: y
Error: Failed to create managed resources: Unable to locate credentials

I need to get my AWS credentials sorted. I'm going to follow https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html and https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-set-up-credentials.html

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645051370 https://github.com/simonw/datasette/issues/850#issuecomment-645051370 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA1MTM3MA== simonw 9599 2020-06-16T22:53:05Z 2020-06-16T22:53:05Z OWNER
simon@Simons-MacBook-Pro datasette-proof-of-concept % sam local invoke
Invoking app.lambda_handler (python3.8)

Fetching lambci/lambda:python3.8 Docker container image....................................................................................................................................................................................................................................
Mounting /private/tmp/datasette-proof-of-concept/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: 4616ab43-6882-1627-e5e3-5a29730d52f9 Version: $LATEST
END RequestId: 4616ab43-6882-1627-e5e3-5a29730d52f9
REPORT RequestId: 4616ab43-6882-1627-e5e3-5a29730d52f9  Init Duration: 140.84 ms    Duration: 2.49 ms   Billed Duration: 100 ms Memory Size: 128 MBMax Memory Used: 25 MB   

{"statusCode":200,"body":"{\"message\": \"hello world\"}"}
simon@Simons-MacBook-Pro datasette-proof-of-concept % sam local invoke
Invoking app.lambda_handler (python3.8)

Fetching lambci/lambda:python3.8 Docker container image......
Mounting /private/tmp/datasette-proof-of-concept/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: 3189df2f-e9c0-1be4-b9ac-f329c5fcd067 Version: $LATEST
END RequestId: 3189df2f-e9c0-1be4-b9ac-f329c5fcd067
REPORT RequestId: 3189df2f-e9c0-1be4-b9ac-f329c5fcd067  Init Duration: 87.22 ms Duration: 2.34 ms   Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 25 MB  

{"statusCode":200,"body":"{\"message\": \"hello world\"}"}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645050948 https://github.com/simonw/datasette/issues/850#issuecomment-645050948 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA1MDk0OA== simonw 9599 2020-06-16T22:51:30Z 2020-06-16T22:52:30Z OWNER
simon@Simons-MacBook-Pro datasette-proof-of-concept % sam build --use-container
Starting Build inside a container
Building function 'HelloWorldFunction'

Fetching lambci/lambda:build-python3.8 Docker container image..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Mounting /private/tmp/datasette-proof-of-concept/hello_world as /tmp/samcli/source:ro,delegated inside runtime container

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645048062 https://github.com/simonw/datasette/issues/850#issuecomment-645048062 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA0ODA2Mg== simonw 9599 2020-06-16T22:41:33Z 2020-06-16T22:41:33Z OWNER
simon@Simons-MacBook-Pro /tmp % sam init

    SAM CLI now collects telemetry to better understand customer needs.

    You can OPT OUT and disable telemetry collection by setting the
    environment variable SAM_CLI_TELEMETRY=0 in your shell.
    Thanks for your help!

    Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html

Which template source would you like to use?
    1 - AWS Quick Start Templates
    2 - Custom Template Location
Choice: 1

Which runtime would you like to use?
    1 - nodejs12.x
    2 - python3.8
    3 - ruby2.7
    4 - go1.x
    5 - java11
    6 - dotnetcore3.1
    7 - nodejs10.x
    8 - python3.7
    9 - python3.6
    10 - python2.7
    11 - ruby2.5
    12 - java8
    13 - dotnetcore2.1
Runtime: 2

Project name [sam-app]: datasette-proof-of-concept

Cloning app templates from https://github.com/awslabs/aws-sam-cli-app-templates.git

AWS quick start application templates:
    1 - Hello World Example
    2 - EventBridge Hello World
    3 - EventBridge App from scratch (100+ Event Schemas)
    4 - Step Functions Sample App (Stock Trader)
Template selection: 1

-----------------------
Generating application:
-----------------------
Name: datasette-proof-of-concept
Runtime: python3.8
Dependency Manager: pip
Application Template: hello-world
Output Directory: .

Next steps can be found in the README file at ./datasette-proof-of-concept/README.md
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645047703 https://github.com/simonw/datasette/issues/850#issuecomment-645047703 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA0NzcwMw== simonw 9599 2020-06-16T22:40:19Z 2020-06-16T22:40:19Z OWNER

Installed SAM:

brew tap aws/tap
brew install aws-sam-cli
sam --version
SAM CLI, version 0.52.0
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645045055 https://github.com/simonw/datasette/issues/850#issuecomment-645045055 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA0NTA1NQ== simonw 9599 2020-06-16T22:31:49Z 2020-06-16T22:31:49Z OWNER

It looks like SAM - AWS Serverless Application Model - is the currently recommended way to deploy Python apps to Lambda from the command-line: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-hello-world.html

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645042625 https://github.com/simonw/datasette/issues/850#issuecomment-645042625 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA0MjYyNQ== simonw 9599 2020-06-16T22:24:26Z 2020-06-16T22:24:26Z OWNER

From https://mangum.io/adapter/

The AWS Lambda handler event and context arguments are made available to an ASGI application in the ASGI connection scope.

scope['aws.event'] scope['aws.context']
I can use https://github.com/simonw/datasette-debug-asgi to see that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645041663 https://github.com/simonw/datasette/issues/850#issuecomment-645041663 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTA0MTY2Mw== simonw 9599 2020-06-16T22:21:44Z 2020-06-16T22:21:44Z OWNER

https://github.com/jordaneremieff/mangum looks like the best way to run an ASGI app on Lambda at the moment.

from mangum import Mangum

async def app(scope, receive, send):
    await send(
        {
            "type": "http.response.start",
            "status": 200,
            "headers": [[b"content-type", b"text/plain; charset=utf-8"]],
        }
    )
    await send({"type": "http.response.body", "body": b"Hello, world!"})


handler = Mangum(app)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645032643 https://github.com/simonw/datasette/issues/850#issuecomment-645032643 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTAzMjY0Mw== simonw 9599 2020-06-16T21:57:10Z 2020-06-16T21:57:10Z OWNER

https://docs.aws.amazon.com/efs/latest/ug/wt1-getting-started.html is an EFS walk-through using the AWS CLI tool instead of clicking around in their web interface.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645031225 https://github.com/simonw/datasette/issues/850#issuecomment-645031225 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTAzMTIyNQ== simonw 9599 2020-06-16T21:53:25Z 2020-06-16T21:53:25Z OWNER

Easier solution to this might be to have two functions - a "read-only" one which is allowed to scale as much as it likes, and a "write-only" one which can write to the database files but is limited to running a maximum of one Lambda instance. https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467
645030262 https://github.com/simonw/datasette/issues/850#issuecomment-645030262 https://api.github.com/repos/simonw/datasette/issues/850 MDEyOklzc3VlQ29tbWVudDY0NTAzMDI2Mg== simonw 9599 2020-06-16T21:51:01Z 2020-06-16T21:51:39Z OWNER

File locking is interesting here. https://docs.aws.amazon.com/lambda/latest/dg/services-efs.html

Amazon EFS supports file locking to prevent corruption if multiple functions try to write to the same file system at the same time. Locking in Amazon EFS follows the NFS v4.1 protocol for advisory locking, and enables your applications to use both whole file and byte range locks.

SQLite can apparently work on NFS v4.1. I think I'd rather set things up so there's only ever one writer - so a Datasette instance could scale reads by running lots more lambda functions but only one function ever writes to a file at a time. Not sure if that's feasible with Lambda though - maybe by adding some additional shared state mechanism like Redis?

{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 1,
    "rocket": 0,
    "eyes": 0
}
Proof of concept for Datasette on AWS Lambda with EFS 639993467

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 27.38ms · About: github-to-sqlite