5,279 rows sorted by updated_at descending

View and edit SQL

issue

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
778815740 https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778815740 https://api.github.com/repos/simonw/sqlite-utils/issues/230 MDEyOklzc3VlQ29tbWVudDc3ODgxNTc0MA== simonw 9599 2021-02-14T18:05:03Z 2021-02-14T18:05:03Z OWNER

The challenge here is how to read the first 2048 bytes and then reset the incoming file.

The Python docs example looks like this:

with open('example.csv', newline='') as csvfile:
    dialect = csv.Sniffer().sniff(csvfile.read(1024))
    csvfile.seek(0)
    reader = csv.reader(csvfile, dialect)

Here's the relevant code in sqlite-utils: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L671-L679

The challenge is going to be having the --sniff option work with the progress bar. Here's how file_progress() works: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/utils.py#L106-L113

If file.raw is stdin can I do the equivalent of csvfile.seek(0) on it?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
--sniff option for sniffing delimiters 808008305  
778812684 https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778812684 https://api.github.com/repos/simonw/sqlite-utils/issues/230 MDEyOklzc3VlQ29tbWVudDc3ODgxMjY4NA== simonw 9599 2021-02-14T17:45:16Z 2021-02-14T17:45:16Z OWNER

Running this could take any CSV (or TSV) file and automatically detect the delimiter. If no header row is detected it could add unknown1,unknown2 headers:

sqlite-utils insert db.db data file.csv --sniff

(Using --sniff would imply --csv)

This could be called --sniffer instead but I like --sniff better.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
--sniff option for sniffing delimiters 808008305  
778812050 https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778812050 https://api.github.com/repos/simonw/sqlite-utils/issues/228 MDEyOklzc3VlQ29tbWVudDc3ODgxMjA1MA== simonw 9599 2021-02-14T17:41:30Z 2021-02-14T17:41:30Z OWNER

I just spotted that csv.Sniffer in the Python standard library has a .has_header(sample) method which detects if the first row appears to be a header or not, which is interesting. https://docs.python.org/3/library/csv.html#csv.Sniffer

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
--no-headers option for CSV and TSV 807437089  
778811934 https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778811934 https://api.github.com/repos/simonw/sqlite-utils/issues/228 MDEyOklzc3VlQ29tbWVudDc3ODgxMTkzNA== simonw 9599 2021-02-14T17:40:48Z 2021-02-14T17:40:48Z OWNER

Another pattern that might be useful is to generate a header that is just "unknown1,unknown2,unknown3" for each of the columns in the rest of the file. This makes it easy to e.g. facet-explore within Datasette to figure out the correct names, then use sqlite-utils transform --rename to rename the columns.

I needed to do that for the https://bl.iro.bl.uk/work/ns/3037474a-761c-456d-a00c-9ef3c6773f4c example.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
--no-headers option for CSV and TSV 807437089  
778511347 https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778511347 https://api.github.com/repos/simonw/sqlite-utils/issues/228 MDEyOklzc3VlQ29tbWVudDc3ODUxMTM0Nw== simonw 9599 2021-02-12T23:27:50Z 2021-02-12T23:27:50Z OWNER

For the moment, a workaround can be to cat an additional row onto the start of the file.

echo "name,url,description" | cat - missing_headings.csv | sqlite-utils insert blah.db table - --csv
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
--no-headers option for CSV and TSV 807437089  
778510528 https://github.com/simonw/sqlite-utils/issues/131#issuecomment-778510528 https://api.github.com/repos/simonw/sqlite-utils/issues/131 MDEyOklzc3VlQ29tbWVudDc3ODUxMDUyOA== simonw 9599 2021-02-12T23:25:06Z 2021-02-12T23:25:06Z OWNER

If -c isn't available, maybe -t or --type would work for specifying column types:

sqlite-utils insert db.db images images.tsv \
  --tsv \
  --type id int \
  --type score float

or

sqlite-utils insert db.db images images.tsv \
  --tsv \
  -t id int \
  -t score float
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
sqlite-utils insert: options for column types 675753042  
778508887 https://github.com/simonw/sqlite-utils/issues/131#issuecomment-778508887 https://api.github.com/repos/simonw/sqlite-utils/issues/131 MDEyOklzc3VlQ29tbWVudDc3ODUwODg4Nw== simonw 9599 2021-02-12T23:20:11Z 2021-02-12T23:20:11Z OWNER

Annoyingly -c is currently a shortcut for --csv - so I'd have to do a major version bump to use that.

https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L601-L603

Particularly annoying because I attempted to remove the -c shortcut in https://github.com/simonw/sqlite-utils/commit/2c00567aac6d9c79087cfff0d054f64922b1473d#diff-76294b3d4afeb27e74e738daa01c26dd4dc9ccb6f4477451483a2ece1095902eL48 but forgot to remove it from the input options (I removed it from the output options).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
sqlite-utils insert: options for column types 675753042  
778467759 https://github.com/simonw/datasette/issues/1220#issuecomment-778467759 https://api.github.com/repos/simonw/datasette/issues/1220 MDEyOklzc3VlQ29tbWVudDc3ODQ2Nzc1OQ== aborruso 30607 2021-02-12T21:35:17Z 2021-02-12T21:35:17Z NONE

Thank you

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Installing datasette via docker: Path 'fixtures.db' does not exist 806743116  
778439617 https://github.com/simonw/datasette/issues/1220#issuecomment-778439617 https://api.github.com/repos/simonw/datasette/issues/1220 MDEyOklzc3VlQ29tbWVudDc3ODQzOTYxNw== bobwhitelock 7476523 2021-02-12T20:33:27Z 2021-02-12T20:33:27Z NONE

That Docker command will mount your current directory inside the Docker container at /mnt - so you shouldn't need to change anything locally, just run

docker run -p 8001:8001 -v `pwd`:/mnt \
    datasetteproject/datasette \
    datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db

and it will use the fixtures.db file within your current directory

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Installing datasette via docker: Path 'fixtures.db' does not exist 806743116  
770069864 https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770069864 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60 MDEyOklzc3VlQ29tbWVudDc3MDA2OTg2NA== daniel-butler 22578954 2021-01-29T21:52:05Z 2021-02-12T18:29:43Z NONE

For the purposes below I am assuming the organization I would get all the repositories and their related commits from is called gh-organization. The github's owner id of gh-orgnization is 123456789.

github-to-sqlite  repos github.db gh-organization

I'm on a windows computer running git bash to be able to use the | command. This works for me

sqlite3 github.db "SELECT full_name FROM repos WHERE owner = '123456789';" | tr '\n\r' ' ' | xargs | { read repos; github-to-sqlite commits github.db $repos; }

On a pure linux system I think this would work because the new line character is normally \n

sqlite3 github.db "SELECT full_name FROM repos WHERE owner = '123456789';" | tr '\n' ' ' | xargs | { read repos; github-to-sqlite commits github.db $repos; }`

As expected I ran into rate limit issues #51

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use Data from SQLite in other commands 797097140  
778349672 https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778349672 https://api.github.com/repos/simonw/sqlite-utils/issues/228 MDEyOklzc3VlQ29tbWVudDc3ODM0OTY3Mg== simonw 9599 2021-02-12T18:00:43Z 2021-02-12T18:00:43Z OWNER

I could combine this with #131 to allow types to be specified in addition to column names.

Probably need an option that means "ignore the existing heading row and use this one instead".

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
--no-headers option for CSV and TSV 807437089  
778349142 https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778349142 https://api.github.com/repos/simonw/sqlite-utils/issues/227 MDEyOklzc3VlQ29tbWVudDc3ODM0OTE0Mg== simonw 9599 2021-02-12T17:59:35Z 2021-02-12T17:59:35Z OWNER

It looks like I can at least bump this size limit up to the maximum allowed by Python - I'll take a look at that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Error reading csv files with large column data 807174161  
778246347 https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778246347 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33 MDEyOklzc3VlQ29tbWVudDc3ODI0NjM0Nw== RhetTbull 41546558 2021-02-12T15:00:43Z 2021-02-12T15:00:43Z CONTRIBUTOR

Yes, Big Sur Photos database doesn't have ZGENERICASSET table. PR #31 will fix this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
photo-to-sqlite: command not found 803338729  
778014990 https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778014990 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33 MDEyOklzc3VlQ29tbWVudDc3ODAxNDk5MA== leafgarland 675335 2021-02-12T06:54:14Z 2021-02-12T06:54:14Z NONE

Ahh, that might be because macOS Big Sur has changed the structure of the photos db. Might need to wait for a later release, there is a PR which adds support for Big Sur.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
photo-to-sqlite: command not found 803338729  
778008752 https://github.com/simonw/datasette/issues/1220#issuecomment-778008752 https://api.github.com/repos/simonw/datasette/issues/1220 MDEyOklzc3VlQ29tbWVudDc3ODAwODc1Mg== aborruso 30607 2021-02-12T06:37:34Z 2021-02-12T06:37:34Z NONE

I have used my path, I'm running it from the folder in wich I have the db.

Do I must an absolute path?

Do I must create exactly that folder?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Installing datasette via docker: Path 'fixtures.db' does not exist 806743116  
778002092 https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778002092 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33 MDEyOklzc3VlQ29tbWVudDc3ODAwMjA5Mg== robmarkcole 11855322 2021-02-12T06:19:32Z 2021-02-12T06:19:32Z NONE

hi @leafgarland that results in a new error:

(venv) (base) Robins-MacBook:datasette robin$ dogsheep-photos apple-photos photos.db
Traceback (most recent call last):
  File "/Users/robin/datasette/venv/bin/dogsheep-photos", line 8, in <module>
    sys.exit(cli())
  File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/cli.py", line 206, in apple_photos
    db.conn.execute(
sqlite3.OperationalError: no such table: attached.ZGENERICASSET
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
photo-to-sqlite: command not found 803338729  
777951854 https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-777951854 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33 MDEyOklzc3VlQ29tbWVudDc3Nzk1MTg1NA== leafgarland 675335 2021-02-12T03:54:39Z 2021-02-12T03:54:39Z NONE

I think that is a typo in the docs, you can use

> dogsheep-photos apple-photos photos.db
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
photo-to-sqlite: command not found 803338729  
777949755 https://github.com/simonw/datasette/pull/1223#issuecomment-777949755 https://api.github.com/repos/simonw/datasette/issues/1223 MDEyOklzc3VlQ29tbWVudDc3Nzk0OTc1NQ== codecov[bot] 22429695 2021-02-12T03:45:31Z 2021-02-12T03:45:31Z NONE

Codecov Report

Merging #1223 (d1cd1f2) into main (9603d89) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1223   +/-   ##
=======================================
  Coverage   91.42%   91.42%           
=======================================
  Files          32       32           
  Lines        3955     3955           
=======================================
  Hits         3616     3616           
  Misses        339      339           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9603d89...d1cd1f2. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add compile option to Dockerfile to fix failing test (fixes #696) 806918878  
777927946 https://github.com/simonw/datasette/issues/1220#issuecomment-777927946 https://api.github.com/repos/simonw/datasette/issues/1220 MDEyOklzc3VlQ29tbWVudDc3NzkyNzk0Ng== bobwhitelock 7476523 2021-02-12T02:29:54Z 2021-02-12T02:29:54Z NONE

According to https://github.com/simonw/datasette/blob/master/docs/installation.rst#using-docker it should be

docker run -p 8001:8001 -v `pwd`:/mnt \
    datasetteproject/datasette \
    datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db

This uses /mnt/fixtures.db whereas you're using fixtures.db - did you try using this path instead?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Installing datasette via docker: Path 'fixtures.db' does not exist 806743116  
777901052 https://github.com/simonw/datasette/issues/1221#issuecomment-777901052 https://api.github.com/repos/simonw/datasette/issues/1221 MDEyOklzc3VlQ29tbWVudDc3NzkwMTA1Mg== simonw 9599 2021-02-12T01:09:54Z 2021-02-12T01:09:54Z OWNER

I also tested this manually. I generated certificate files like so:

cd /tmp
python -m trustme

This created /tmp/server.pem, /tmp/client.pem and /tmp/server.key

Then I started Datasette like this:

datasette --memory --ssl-keyfile=/tmp/server.key --ssl-certfile=/tmp/server.pem

And exercise it using curl like so:

/tmp % curl --cacert /tmp/client.pem 'https://localhost:8001/_memory.json'
{"database": "_memory", "path": "/_memory", "size": 0, "tables": [], "hidden_count": 0, "views": [], "queries": [],
"private": false, "allow_execute_sql": true, "query_ms": 0.8843200000114848}

Note that without the --cacert option I get an error:

```
/tmp % curl 'https://localhost:8001/_memory.json'
curl: (60) SSL certificate problem: Invalid certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support SSL/TLS directly 806849424  
777887190 https://github.com/simonw/datasette/issues/1221#issuecomment-777887190 https://api.github.com/repos/simonw/datasette/issues/1221 MDEyOklzc3VlQ29tbWVudDc3Nzg4NzE5MA== simonw 9599 2021-02-12T00:29:18Z 2021-02-12T00:29:18Z OWNER

I can use this recipe to start a datasette server in a sub-process during the pytest run and exercise it with real HTTP requests: https://til.simonwillison.net/pytest/subprocess-server

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support SSL/TLS directly 806849424  
777883452 https://github.com/simonw/datasette/issues/1221#issuecomment-777883452 https://api.github.com/repos/simonw/datasette/issues/1221 MDEyOklzc3VlQ29tbWVudDc3Nzg4MzQ1Mg== simonw 9599 2021-02-12T00:19:30Z 2021-02-12T00:19:40Z OWNER

Uvicorn supports these options: https://www.uvicorn.org/#command-line-options

  --ssl-keyfile TEXT              SSL key file
  --ssl-certfile TEXT             SSL certificate file
  --ssl-keyfile-password TEXT     SSL keyfile password
  --ssl-version INTEGER           SSL version to use (see stdlib ssl module's)
                                  [default: 2]

  --ssl-cert-reqs INTEGER         Whether client certificate is required (see
                                  stdlib ssl module's)  [default: 0]

  --ssl-ca-certs TEXT             CA certificates file
  --ssl-ciphers TEXT              Ciphers to use (see stdlib ssl module's)
                                  [default: TLSv1]

For the moment I'm going to support just --ssl-keyfile and --ssl-certfile as arguments to datasette serve. I'll add other options if people ask for them.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support SSL/TLS directly 806849424  
777839351 https://github.com/dogsheep/evernote-to-sqlite/pull/10#issuecomment-777839351 https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10 MDEyOklzc3VlQ29tbWVudDc3NzgzOTM1MQ== simonw 9599 2021-02-11T22:37:55Z 2021-02-11T22:37:55Z MEMBER

I've merged these changes by hand now, thanks!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
BugFix for encoding and not update info. 770712149  
777827396 https://github.com/dogsheep/evernote-to-sqlite/issues/7#issuecomment-777827396 https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7 MDEyOklzc3VlQ29tbWVudDc3NzgyNzM5Ng== simonw 9599 2021-02-11T22:13:14Z 2021-02-11T22:13:14Z MEMBER

My best guess is that you have an older version of sqlite-utils installed here - the replace=True argument was added in version 2.0. I've bumped the dependency in setup.py.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace' 743297582  
777821383 https://github.com/dogsheep/evernote-to-sqlite/issues/9#issuecomment-777821383 https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9 MDEyOklzc3VlQ29tbWVudDc3NzgyMTM4Mw== simonw 9599 2021-02-11T22:01:28Z 2021-02-11T22:01:28Z MEMBER

Aha! I think I've figured out what's going on here.

The CData blocks containing the notes look like this:

<![CDATA[<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd"><en-note><div>This note includes two images.</div><div><br /></div>...

The DTD at http://xml.evernote.com/pub/enml2.dtd includes some entities:

<!--=========== External character mnemonic entities ===================-->

<!ENTITY % HTMLlat1 PUBLIC
   "-//W3C//ENTITIES Latin 1 for XHTML//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent">
%HTMLlat1;

<!ENTITY % HTMLsymbol PUBLIC
   "-//W3C//ENTITIES Symbols for XHTML//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent">
%HTMLsymbol;

<!ENTITY % HTMLspecial PUBLIC
   "-//W3C//ENTITIES Special for XHTML//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent">
%HTMLspecial;

So I need to be able to handle all of those different entities. I think I can do that using html.entities.entitydefs from the Python standard library, which looks a bit like this:

{'Aacute': 'Á',
 'aacute': 'á',
 'Aacute;': 'Á',
 'aacute;': 'á',
 'Abreve;': 'Ă',
 'abreve;': 'ă',
 'ac;': '∾',
 'acd;': '∿',
# ...
}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
ParseError: undefined entity &scaron; 748372469  
777798330 https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777798330 https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11 MDEyOklzc3VlQ29tbWVudDc3Nzc5ODMzMA== simonw 9599 2021-02-11T21:18:58Z 2021-02-11T21:18:58Z MEMBER

Thanks for the fix!

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
XML parse error 792851444  
777690332 https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777690332 https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11 MDEyOklzc3VlQ29tbWVudDc3NzY5MDMzMg== dskrad 3613583 2021-02-11T18:16:01Z 2021-02-11T18:16:01Z NONE

I solved this issue by modifying line 31 of utils.py

content = ET.tostring(ET.fromstring(content_xml.strip())).decode("utf-8")
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
XML parse error 792851444  
777178728 https://github.com/simonw/datasette/issues/1200#issuecomment-777178728 https://api.github.com/repos/simonw/datasette/issues/1200 MDEyOklzc3VlQ29tbWVudDc3NzE3ODcyOA== simonw 9599 2021-02-11T03:13:59Z 2021-02-11T03:13:59Z OWNER

I came up with the need for this while playing with this tool: https://calands.datasettes.com/calands?sql=select%0D%0A++AsGeoJSON(geometry)%2C+*%0D%0Afrom%0D%0A++CPAD_2020a_SuperUnits%0D%0Awhere%0D%0A++PARK_NAME+like+'%25mini%25'+and%0D%0A++Intersects(GeomFromGeoJSON(%3Afreedraw)%2C+geometry)+%3D+1%0D%0A++and+CPAD_2020a_SuperUnits.rowid+in+(%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++SpatialIndex%0D%0A++++where%0D%0A++++++f_table_name+%3D+'CPAD_2020a_SuperUnits'%0D%0A++++++and+search_frame+%3D+GeomFromGeoJSON(%3Afreedraw)%0D%0A++)&freedraw={"type"%3A"MultiPolygon"%2C"coordinates"%3A[[[[-122.42202758789064%2C37.82280243352759]%2C[-122.39868164062501%2C37.823887203271454]%2C[-122.38220214843751%2C37.81846319511331]%2C[-122.35061645507814%2C37.77071473849611]%2C[-122.34924316406251%2C37.74465712069939]%2C[-122.37258911132814%2C37.703380457832374]%2C[-122.39044189453125%2C37.690340943717715]%2C[-122.41241455078126%2C37.680559803205135]%2C[-122.44262695312501%2C37.67295135774715]%2C[-122.47283935546876%2C37.67295135774715]%2C[-122.52502441406251%2C37.68382032669382]%2C[-122.53463745117189%2C37.6892542140253]%2C[-122.54699707031251%2C37.690340943717715]%2C[-122.55798339843751%2C37.72945260537781]%2C[-122.54287719726564%2C37.77831314799672]%2C[-122.49893188476564%2C37.81303878836991]%2C[-122.46185302734376%2C37.82822612280363]%2C[-122.42889404296876%2C37.82822612280363]%2C[-122.42202758789064%2C37.82280243352759]]]]} - before I fixed https://github.com/simonw/datasette-leaflet-geojson/issues/16 it was loading a LOT of maps, which felt bad. I wanted to be able to link people to that page with a hard limit on the number of rows displayed on that page.

It's mainly to guard against unexpected behaviour from limit-less queries though. It's not a very high priority feature!

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_size=10 option for the arbitrary query page would be useful 792890765  
777132761 https://github.com/simonw/datasette/issues/1200#issuecomment-777132761 https://api.github.com/repos/simonw/datasette/issues/1200 MDEyOklzc3VlQ29tbWVudDc3NzEzMjc2MQ== bobwhitelock 7476523 2021-02-11T00:29:52Z 2021-02-11T00:29:52Z NONE

I'm probably missing something but what's the use case here - what would this offer over adding limit 10 to the query?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_size=10 option for the arbitrary query page would be useful 792890765  
775442039 https://github.com/simonw/datasette/issues/1219#issuecomment-775442039 https://api.github.com/repos/simonw/datasette/issues/1219 MDEyOklzc3VlQ29tbWVudDc3NTQ0MjAzOQ== simonw 9599 2021-02-08T20:39:52Z 2021-02-08T22:13:00Z OWNER

This comment helped me find a pattern for running Scalene against the Datasette test suite: https://github.com/emeryberger/scalene/issues/70#issuecomment-755245858

pip install scalene

Then I created a file called run_tests.py with the following contents:

if __name__ == "__main__":
    import sys, pytest
    pytest.main(sys.argv)

Then I ran this:

scalene --profile-all run_tests.py -sv -x .

But... it quit with a segmentation fault!

(datasette) datasette % scalene --profile-all run_tests.py -sv -x .
======================================================================== test session starts ========================================================================
platform darwin -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- python
cachedir: .pytest_cache
rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
plugins: asyncio-0.14.0, timeout-1.4.2
collecting ... Fatal Python error: Segmentation fault

Current thread 0x0000000110c1edc0 (most recent call first):
  File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 553 in detect_json1
  File "/Users/simon/Dropbox/Development/datasette/datasette/filters.py", line 168 in Filters
  File "/Users/simon/Dropbox/Development/datasette/datasette/filters.py", line 94 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 27 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 42 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/Users/simon/Dropbox/Development/datasette/tests/test_api.py", line 1 in <module>
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 170 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1014 in _gcd_import
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127 in import_module
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/pathlib.py", line 520 in import_path
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py", line 552 in _importtestmodule
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py", line 484 in _getobj
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py", line 288 in obj
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py", line 500 in _inject_setup_module_fixture
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py", line 487 in collect
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py", line 324 in <lambda>
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py", line 294 in from_call
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py", line 324 in pytest_make_collect_report
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py", line 441 in collect_one_node
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 768 in genitems
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 771 in genitems
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 568 in _perform_collect
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 516 in perform_collect
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 306 in pytest_collection
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 295 in _main
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 240 in wrap_session
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 289 in pytest_cmdline_main
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/config/__init__.py", line 157 in main
  File "run_tests.py", line 3 in <module>
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/scalene/scalene_profiler.py", line 1525 in main
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/scalene/__main__.py", line 7 in main
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/scalene/__main__.py", line 14 in <module>
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87 in _run_code
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194 in _run_module_as_main
Scalene error: received signal SIGSEGV
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Try profiling Datasette using scalene 803929694  
775497449 https://github.com/simonw/datasette/issues/1219#issuecomment-775497449 https://api.github.com/repos/simonw/datasette/issues/1219 MDEyOklzc3VlQ29tbWVudDc3NTQ5NzQ0OQ== simonw 9599 2021-02-08T22:11:34Z 2021-02-08T22:11:34Z OWNER

https://github.com/emeryberger/scalene/issues/110 reports a "received signal SIGSEGV" error that was fixed by upgrading to the latest Scalene version, but I'm running that already.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Try profiling Datasette using scalene 803929694  
774730656 https://github.com/dogsheep/pocket-to-sqlite/issues/9#issuecomment-774730656 https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9 MDEyOklzc3VlQ29tbWVudDc3NDczMDY1Ng== merwok 635179 2021-02-07T18:45:04Z 2021-02-07T18:45:04Z NONE

That URL uses TLS 1.3, but maybe only if the client supports it.
It could be your Python version or your SSL library that’s not recent enough.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
SSL Error 801780625  
774726123 https://github.com/dogsheep/pocket-to-sqlite/issues/9#issuecomment-774726123 https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9 MDEyOklzc3VlQ29tbWVudDc3NDcyNjEyMw== jfeiwell 12669260 2021-02-07T18:21:08Z 2021-02-07T18:21:08Z NONE

@simonw any ideas here?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
SSL Error 801780625  
774528913 https://github.com/simonw/datasette/issues/1217#issuecomment-774528913 https://api.github.com/repos/simonw/datasette/issues/1217 MDEyOklzc3VlQ29tbWVudDc3NDUyODkxMw== virtadpt 639730 2021-02-06T19:23:41Z 2021-02-06T19:23:41Z NONE

I've had a lot of success running it as an OpenFaaS lambda.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Possible to deploy as a python app (for Rstudio connect server)? 802513359  
774385092 https://github.com/simonw/datasette/issues/1217#issuecomment-774385092 https://api.github.com/repos/simonw/datasette/issues/1217 MDEyOklzc3VlQ29tbWVudDc3NDM4NTA5Mg== pavopax 6165713 2021-02-06T02:49:11Z 2021-02-06T02:49:11Z NONE

A good reference seems to be the note to run datasette as a module in https://github.com/simonw/datasette/pull/556

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Possible to deploy as a python app (for Rstudio connect server)? 802513359  
774373829 https://github.com/simonw/sqlite-utils/issues/223#issuecomment-774373829 https://api.github.com/repos/simonw/sqlite-utils/issues/223 MDEyOklzc3VlQ29tbWVudDc3NDM3MzgyOQ== simonw 9599 2021-02-06T01:39:47Z 2021-02-06T01:39:47Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
--delimiter option for CSV import 788527932  
774286962 https://github.com/simonw/datasette/issues/1208#issuecomment-774286962 https://api.github.com/repos/simonw/datasette/issues/1208 MDEyOklzc3VlQ29tbWVudDc3NDI4Njk2Mg== kbaikov 4488943 2021-02-05T21:02:39Z 2021-02-05T21:02:39Z NONE

@simonw could you please take a look at the PR 1211 that fixes this issue?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
A lot of open(file) functions are used without a context manager thus producing ResourceWarning: unclosed file <_io.TextIOWrapper 794554881  
774217792 https://github.com/simonw/sqlite-utils/pull/203#issuecomment-774217792 https://api.github.com/repos/simonw/sqlite-utils/issues/203 MDEyOklzc3VlQ29tbWVudDc3NDIxNzc5Mg== drkane 1049910 2021-02-05T18:44:13Z 2021-02-05T18:44:13Z NONE

Thanks for looking at this - home schooling kids has prevented me from replying.

I'd struggled with how to adapt the API for the foreign keys too - I definitely tried the String/Tuple approach. I hadn't considered the breaking changes that would introduce though. I can take a look at this and try and make the change - see which of your options works best.

I've got a workaround for the use-case I was looking at this for, so it wouldn't be a problem for me if it was put on the back burner until a hypothetical v4.0 anyway.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
changes to allow for compound foreign keys 743384829  
773977128 https://github.com/simonw/datasette/issues/1210#issuecomment-773977128 https://api.github.com/repos/simonw/datasette/issues/1210 MDEyOklzc3VlQ29tbWVudDc3Mzk3NzEyOA== heyarne 525780 2021-02-05T11:30:34Z 2021-02-05T11:30:34Z NONE

Thanks for your quick reply! Having changed my metadata.yml, queries AND database I can't really reproduce it anymore, sorry. But at least I'm happy to say that it works now! :) Thanks again for the super nifty tool, very appreciated.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Immutable Database w/ Canned Queries 796234313  
772796111 https://github.com/simonw/datasette/issues/1216#issuecomment-772796111 https://api.github.com/repos/simonw/datasette/issues/1216 MDEyOklzc3VlQ29tbWVudDc3Mjc5NjExMQ== simonw 9599 2021-02-03T20:20:48Z 2021-02-03T20:20:48Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
/-/databases should reflect connection order, not alphabetical order 800669347  
772408273 https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-772408273 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56 MDEyOklzc3VlQ29tbWVudDc3MjQwODI3Mw== gsajko 42315895 2021-02-03T10:36:36Z 2021-02-03T10:36:36Z NONE

I figured it out.
Those tweets are in database, because somebody quote tweeted them, or retweeted them.
And if you grab quoted tweet or reweeted tweet from other tweet json, It doesn't grab all of the details.

So if someone quote tweeted a quote tweet, the second quote tweet won't have quoted_status.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Not all quoted statuses get fetched? 796736607  
772007663 https://github.com/simonw/datasette/issues/1212#issuecomment-772007663 https://api.github.com/repos/simonw/datasette/issues/1212 MDEyOklzc3VlQ29tbWVudDc3MjAwNzY2Mw== kbaikov 4488943 2021-02-02T21:36:56Z 2021-02-02T21:36:56Z NONE

How do you get 4-5 minutes?
I run my tests in WSL 2, so may be i need to try a real linux VM.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Tests are very slow.  797651831  
772001787 https://github.com/simonw/datasette/issues/1214#issuecomment-772001787 https://api.github.com/repos/simonw/datasette/issues/1214 MDEyOklzc3VlQ29tbWVudDc3MjAwMTc4Nw== simonw 9599 2021-02-02T21:28:53Z 2021-02-02T21:28:53Z OWNER

Fix is now live on https://latest.datasette.io/fixtures/searchable?_search=terry - clearing "terry" and re-submitting the form now works as expected.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Re-submitting filter form duplicates _x querystring arguments 799693777  
771992628 https://github.com/simonw/datasette/issues/1214#issuecomment-771992628 https://api.github.com/repos/simonw/datasette/issues/1214 MDEyOklzc3VlQ29tbWVudDc3MTk5MjYyOA== simonw 9599 2021-02-02T21:15:18Z 2021-02-02T21:15:18Z OWNER

The cause of this bug is form fields which begin with _ but ARE displayed as form inputs on the page - hence should not be duplicated in an <input type="hidden"> element.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Re-submitting filter form duplicates _x querystring arguments 799693777  
771992025 https://github.com/simonw/datasette/issues/1214#issuecomment-771992025 https://api.github.com/repos/simonw/datasette/issues/1214 MDEyOklzc3VlQ29tbWVudDc3MTk5MjAyNQ== simonw 9599 2021-02-02T21:14:16Z 2021-02-02T21:14:16Z OWNER

As a result, navigating to https://github-to-sqlite.dogsheep.net/github/labels?_search=help and clearing out the _search field then submitting the form does NOT clear the search term.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Re-submitting filter form duplicates _x querystring arguments 799693777  
771976561 https://github.com/simonw/datasette/issues/1212#issuecomment-771976561 https://api.github.com/repos/simonw/datasette/issues/1212 MDEyOklzc3VlQ29tbWVudDc3MTk3NjU2MQ== simonw 9599 2021-02-02T20:53:27Z 2021-02-02T20:53:27Z OWNER

It would be great if we could get python-xdist to run too - I tried it in the past and gave up when I ran into those race conditions, but I've not done any further digging to see if there's a way to fix that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Tests are very slow.  797651831  
771975941 https://github.com/simonw/datasette/issues/1212#issuecomment-771975941 https://api.github.com/repos/simonw/datasette/issues/1212 MDEyOklzc3VlQ29tbWVudDc3MTk3NTk0MQ== simonw 9599 2021-02-02T20:52:36Z 2021-02-02T20:52:36Z OWNER

37 minutes, wow! They're a little slow for me (4-5 minutes perhaps) but not nearly that bad.

Thanks for running that profile. I think you're right: figuring out how to use more session scopes would definitely help.

The :memory: idea is interesting too. The new memory_name= feature added in #1151 (released in Datasette 0.54) could help a lot here, since it allows Datasette instances to share the same in-memory database across multiple HTTP requests and connections.

Note that memory_name= also persists within test runs themselves, independently of any scope= options on the fixtures. That might actually help us here!

I'd be delighted if you explored this issue further, especially the option of using memory_name= for the fixtures databases used by the tests.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Tests are very slow.  797651831  
771968675 https://github.com/simonw/datasette/issues/1213#issuecomment-771968675 https://api.github.com/repos/simonw/datasette/issues/1213 MDEyOklzc3VlQ29tbWVudDc3MTk2ODY3NQ== simonw 9599 2021-02-02T20:41:55Z 2021-02-02T20:41:55Z OWNER

So maybe I could a special response header which ASGI middleware can pick up that means "Don't attempt to gzip this, just stream it through".

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
gzip support for HTML (and JSON) responses 799663959  
771968177 https://github.com/simonw/datasette/issues/1213#issuecomment-771968177 https://api.github.com/repos/simonw/datasette/issues/1213 MDEyOklzc3VlQ29tbWVudDc3MTk2ODE3Nw== simonw 9599 2021-02-02T20:41:13Z 2021-02-02T20:41:13Z OWNER

Starlette accumulates the full response body in a body variable and then does this:

        elif message_type == "http.response.body":
            # Remaining body in streaming GZip response.
            body = message.get("body", b"")
            more_body = message.get("more_body", False)

            self.gzip_file.write(body)
            if not more_body:
                self.gzip_file.close()

            message["body"] = self.gzip_buffer.getvalue()
            self.gzip_buffer.seek(0)
            self.gzip_buffer.truncate()

            await self.send(message)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
gzip support for HTML (and JSON) responses 799663959  
771965281 https://github.com/simonw/datasette/issues/1213#issuecomment-771965281 https://api.github.com/repos/simonw/datasette/issues/1213 MDEyOklzc3VlQ29tbWVudDc3MTk2NTI4MQ== simonw 9599 2021-02-02T20:37:08Z 2021-02-02T20:39:24Z OWNER

Starlette's gzip middleware implementation is here: https://github.com/encode/starlette/blob/0.14.2/starlette/middleware/gzip.py

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
gzip support for HTML (and JSON) responses 799663959  
771127458 https://github.com/simonw/datasette/pull/1211#issuecomment-771127458 https://api.github.com/repos/simonw/datasette/issues/1211 MDEyOklzc3VlQ29tbWVudDc3MTEyNzQ1OA== kbaikov 4488943 2021-02-01T20:13:39Z 2021-02-01T20:13:39Z NONE

Ping @simonw

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use context manager instead of plain open 797649915  
770865698 https://github.com/simonw/datasette/pull/1159#issuecomment-770865698 https://api.github.com/repos/simonw/datasette/issues/1159 MDEyOklzc3VlQ29tbWVudDc3MDg2NTY5OA== lovasoa 552629 2021-02-01T13:42:29Z 2021-02-01T13:42:29Z NONE

@simonw : Could you have a look at this ? I think this really improves readability.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improve the display of facets information 774332247  
770343684 https://github.com/simonw/datasette/pull/1211#issuecomment-770343684 https://api.github.com/repos/simonw/datasette/issues/1211 MDEyOklzc3VlQ29tbWVudDc3MDM0MzY4NA== codecov[bot] 22429695 2021-01-31T08:03:40Z 2021-01-31T08:03:40Z NONE

Codecov Report

Merging #1211 (e33ccaa) into main (dde3c50) will decrease coverage by 0.00%.
The diff coverage is 92.85%.

@@            Coverage Diff             @@
##             main    #1211      +/-   ##
==========================================
- Coverage   91.54%   91.53%   -0.01%     
==========================================
  Files          32       32              
  Lines        3948     3959      +11     
==========================================
+ Hits         3614     3624      +10     
- Misses        334      335       +1     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/cli.py</td> <td>77.29% <66.66%> (-0.31%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/app.py</td> <td>95.62% <100.00%> (+<0.01%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/publish/cloudrun.py</td> <td>96.96% <100.00%> (+0.09%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/publish/heroku.py</td> <td>87.73% <100.00%> (+0.60%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/utils/__init__.py</td> <td>94.13% <100.00%> (+0.02%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dde3c50...e33ccaa. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use context manager instead of plain open 797649915  
770150526 https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-770150526 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51 MDEyOklzc3VlQ29tbWVudDc3MDE1MDUyNg== daniel-butler 22578954 2021-01-30T03:44:19Z 2021-01-30T03:47:24Z NONE

I don't have much experience with github's rate limiting. In my day job we use the tenacity library to handle http errors we get.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
github-to-sqlite should handle rate limits better 703246031  
770112248 https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770112248 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60 MDEyOklzc3VlQ29tbWVudDc3MDExMjI0OA== daniel-butler 22578954 2021-01-30T00:01:03Z 2021-01-30T01:14:42Z NONE

Yes that would be cool! I wouldn't mind helping. Is this the meat of it? https://github.com/dogsheep/twitter-to-sqlite/blob/21fc1cad6dd6348c67acff90a785b458d3a81275/twitter_to_sqlite/utils.py#L512

It looks like the cli option is added with this decorator : https://github.com/dogsheep/twitter-to-sqlite/blob/21fc1cad6dd6348c67acff90a785b458d3a81275/twitter_to_sqlite/cli.py#L14

I looked a bit at utils.py in the GitHub repository. I was surprised at the amount of manual mapping of the API response you had to do to get this to work.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use Data from SQLite in other commands 797097140  
770071568 https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770071568 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60 MDEyOklzc3VlQ29tbWVudDc3MDA3MTU2OA== simonw 9599 2021-01-29T21:56:15Z 2021-01-29T21:56:15Z MEMBER

I really like the way you're using pipes here - really smart. It's similar to how I build the demo database in this GitHub Actions workflow:

https://github.com/dogsheep/github-to-sqlite/blob/62dfd3bc4014b108200001ef4bc746feb6f33b45/.github/workflows/deploy-demo.yml#L52-L82

twitter-to-sqlite actually has a mechanism for doing this kind of thing, documented at https://github.com/dogsheep/twitter-to-sqlite#providing-input-from-a-sql-query-with---sql-and---attach

It lets you do things like:

$ twitter-to-sqlite users-lookup my.db --sql="select follower_id from following" --ids

Maybe I should add something similar to github-to-sqlite? Feels like it could be really useful.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use Data from SQLite in other commands 797097140  
769973212 https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769973212 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56 MDEyOklzc3VlQ29tbWVudDc2OTk3MzIxMg== gsajko 42315895 2021-01-29T18:29:02Z 2021-01-29T18:31:55Z NONE

I think it was with twitter-to-sqlite home-timeline home.db -a auth.json --since
and Im using only this command to grab tweets

from cron tab
2,7,12,17,22,27,32,37,42,47,52,57 * * * * run-one /home/gsajko/miniconda3/bin/twitter-to-sqlite home-timeline /home/gsajko/work/custom_twitter_feed/home.db -a /home/gsajko/work/custom_twitter_feed/auth/auth.json --since

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Not all quoted statuses get fetched? 796736607  
769957751 https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769957751 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56 MDEyOklzc3VlQ29tbWVudDc2OTk1Nzc1MQ== simonw 9599 2021-01-29T17:59:40Z 2021-01-29T17:59:40Z MEMBER

This is interesting - how did you create that initial table? Was this using the twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip command, or something else?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Not all quoted statuses get fetched? 796736607  
769534187 https://github.com/simonw/datasette/issues/1207#issuecomment-769534187 https://api.github.com/repos/simonw/datasette/issues/1207 MDEyOklzc3VlQ29tbWVudDc2OTUzNDE4Nw== simonw 9599 2021-01-29T02:37:19Z 2021-01-29T02:37:19Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Document the Datasette(..., pdb=True) testing pattern 793881756  
769455370 https://github.com/simonw/datasette/issues/1209#issuecomment-769455370 https://api.github.com/repos/simonw/datasette/issues/1209 MDEyOklzc3VlQ29tbWVudDc2OTQ1NTM3MA== simonw 9599 2021-01-28T23:00:21Z 2021-01-28T23:00:21Z OWNER

Good catch on the workaround here. The root problem is that datasette-template-sql looks for the first available databsae if you don't provide it with a database= argument, and in Datasette 0.54 the first available database changed to being the new _internal database.

Is this a bug? I think it is - because the documented behaviour on https://docs.datasette.io/en/stable/internals.html#get-database-name is this:

name - string, optional

The name to be used for this database - this will be used in the URL path, e.g. /dbname. If not specified Datasette will pick one based on the filename or memory name.

Since the new behaviour differs from what was in the documentation I'm going to treat this as a bug and fix it.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
v0.54 500 error from sql query in custom template; code worked in v0.53; found a workaround 795367402  
769453074 https://github.com/simonw/datasette/issues/1205#issuecomment-769453074 https://api.github.com/repos/simonw/datasette/issues/1205 MDEyOklzc3VlQ29tbWVudDc2OTQ1MzA3NA== simonw 9599 2021-01-28T22:54:49Z 2021-01-28T22:55:02Z OWNER

I also checked that the following works:

echo '{"foo": "bar"}' | sqlite-utils insert _memory.db demo -
datasette _memory.db --memory

Sure enough, it results in the following Datasette homepage - thanks to #509

https://user-images.githubusercontent.com/9599/106208790-c8564980-6178-11eb-8b8b-053a9f1d0193.png">

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename /:memory: to /_memory 793027837  
769452084 https://github.com/simonw/datasette/issues/1205#issuecomment-769452084 https://api.github.com/repos/simonw/datasette/issues/1205 MDEyOklzc3VlQ29tbWVudDc2OTQ1MjA4NA== simonw 9599 2021-01-28T22:52:23Z 2021-01-28T22:52:23Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename /:memory: to /_memory 793027837  
769442165 https://github.com/simonw/datasette/issues/1205#issuecomment-769442165 https://api.github.com/repos/simonw/datasette/issues/1205 MDEyOklzc3VlQ29tbWVudDc2OTQ0MjE2NQ== simonw 9599 2021-01-28T22:30:16Z 2021-01-28T22:30:27Z OWNER

I'm going to do this, with redirects from /:memory:*.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename /:memory: to /_memory 793027837  
769274591 https://github.com/simonw/datasette/issues/1210#issuecomment-769274591 https://api.github.com/repos/simonw/datasette/issues/1210 MDEyOklzc3VlQ29tbWVudDc2OTI3NDU5MQ== simonw 9599 2021-01-28T18:10:02Z 2021-01-28T18:10:02Z OWNER

That definitely sounds like a bug! Can you provide a copy of your metadata.JSON and the command-line you are using to launch Datasette?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Immutable Database w/ Canned Queries 796234313  
767888743 https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-767888743 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54 MDEyOklzc3VlQ29tbWVudDc2Nzg4ODc0Mw== henry501 19328961 2021-01-26T23:07:41Z 2021-01-26T23:07:41Z NONE

My import got much further with the applied fixes than 0.21.3, but not 100%. I do appear to have all of the tweets imported at least.
Not sure when I'll have a chance to look further to try to fix or see what didn't make it into the import.

Here's my output:

direct-messages-group: not yet implemented
branch-links: not yet implemented
periscope-expired-broadcasts: not yet implemented
direct-messages: not yet implemented
mute: not yet implemented
periscope-comments-made-by-user: not yet implemented
periscope-ban-information: not yet implemented
periscope-profile-description: not yet implemented
screen-name-change: not yet implemented
manifest: not yet implemented
fleet: not yet implemented
user-link-clicks: not yet implemented
periscope-broadcast-metadata: not yet implemented
contact: not yet implemented
fleet-mute: not yet implemented
device-token: not yet implemented
protected-history: not yet implemented
direct-message-mute: not yet implemented
Traceback (most recent call last):
  File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/bin/twitter-to-sqlite", line 33, in <module>
    sys.exit(load_entry_point('twitter-to-sqlite==0.21.3', 'console_scripts', 'twitter-to-sqlite')())
  File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/cli.py", line 772, in import_
    archive.import_from_file(db, filepath.name, open(filepath, "rb").read())
  File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py", line 233, in import_from_file
    to_insert = transformer(data)
  File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py", line 21, in callback
    return {filename: [fn(item) for item in data]}
  File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py", line 21, in <listcomp>
    return {filename: [fn(item) for item in data]}
  File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py", line 88, in ageinfo
    return item["ageMeta"]["ageInfo"]
KeyError: 'ageInfo'
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Archive import appears to be broken on recent exports 779088071  
767823684 https://github.com/simonw/datasette/issues/1208#issuecomment-767823684 https://api.github.com/repos/simonw/datasette/issues/1208 MDEyOklzc3VlQ29tbWVudDc2NzgyMzY4NA== simonw 9599 2021-01-26T20:58:51Z 2021-01-26T20:58:51Z OWNER

This is a good catch - I've been lazy about this, but you're right that it's an issue that needs cleaning up. Would be very happy to apply a PR, thanks!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
A lot of open(file) functions are used without a context manager thus producing ResourceWarning: unclosed file <_io.TextIOWrapper 794554881  
767762551 https://github.com/simonw/datasette/issues/1151#issuecomment-767762551 https://api.github.com/repos/simonw/datasette/issues/1151 MDEyOklzc3VlQ29tbWVudDc2Nzc2MjU1MQ== simonw 9599 2021-01-26T19:07:44Z 2021-01-26T19:07:44Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database class mechanism for cross-connection in-memory databases 770448622  
767761155 https://github.com/simonw/datasette/issues/991#issuecomment-767761155 https://api.github.com/repos/simonw/datasette/issues/991 MDEyOklzc3VlQ29tbWVudDc2Nzc2MTE1NQ== simonw 9599 2021-01-26T19:05:21Z 2021-01-26T19:06:36Z OWNER

Idea: implement this using the existing table view, with a custom template called table-internal-bb0ec0-tables.html - that's the custom template listed in the HTML comments at the bottom of https://latest.datasette.io/_internal/tables

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Redesign application homepage 714377268  
766991680 https://github.com/simonw/datasette/issues/1201#issuecomment-766991680 https://api.github.com/repos/simonw/datasette/issues/1201 MDEyOklzc3VlQ29tbWVudDc2Njk5MTY4MA== simonw 9599 2021-01-25T17:42:21Z 2021-01-25T17:42:21Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Release notes for Datasette 0.54 792904595  
766589070 https://github.com/simonw/datasette/pull/1206#issuecomment-766589070 https://api.github.com/repos/simonw/datasette/issues/1206 MDEyOklzc3VlQ29tbWVudDc2NjU4OTA3MA== codecov[bot] 22429695 2021-01-25T06:50:30Z 2021-01-25T17:31:11Z NONE

Codecov Report

Merging #1206 (06480e1) into main (a5ede3c) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1206   +/-   ##
=======================================
  Coverage   91.53%   91.53%           
=======================================
  Files          32       32           
  Lines        3947     3947           
=======================================
  Hits         3613     3613           
  Misses        334      334           
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/version.py</td> <td>100.00% <100.00%> (ø)</td> <td></td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a5ede3c...571476d. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Release 0.54 793086333  
766588371 https://github.com/simonw/datasette/pull/1206#issuecomment-766588371 https://api.github.com/repos/simonw/datasette/issues/1206 MDEyOklzc3VlQ29tbWVudDc2NjU4ODM3MQ== simonw 9599 2021-01-25T06:49:06Z 2021-01-25T06:49:06Z OWNER

Last thing to do: write up the annotated version of these release notes, assign it a URL on my blog and link to it from the release notes here so I can publish them simultaneously.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Release 0.54 793086333  
766588020 https://github.com/simonw/datasette/pull/1206#issuecomment-766588020 https://api.github.com/repos/simonw/datasette/issues/1206 MDEyOklzc3VlQ29tbWVudDc2NjU4ODAyMA== simonw 9599 2021-01-25T06:48:20Z 2021-01-25T06:48:20Z OWNER

Issues to reference in the commit message: #509, #1091, #1150, #1151, #1166, #1167, #1178, #1181, #1182, #1184, #1185, #1186, #1187, #1194, #1198

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Release 0.54 793086333  
766586151 https://github.com/simonw/datasette/issues/1201#issuecomment-766586151 https://api.github.com/repos/simonw/datasette/issues/1201 MDEyOklzc3VlQ29tbWVudDc2NjU4NjE1MQ== simonw 9599 2021-01-25T06:44:43Z 2021-01-25T06:44:43Z OWNER

OK, release notes are ready to merge from that branch. I'll ship the release in the morning, to give me time to write the accompanying annotated release notes.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Release notes for Datasette 0.54 792904595  
766545604 https://github.com/simonw/datasette/issues/1201#issuecomment-766545604 https://api.github.com/repos/simonw/datasette/issues/1201 MDEyOklzc3VlQ29tbWVudDc2NjU0NTYwNA== simonw 9599 2021-01-25T05:14:31Z 2021-01-25T05:14:31Z OWNER

The two big ticket items are <script type="module"> support and the new _internal mechanism.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Release notes for Datasette 0.54 792904595  
766545442 https://github.com/simonw/datasette/issues/1201#issuecomment-766545442 https://api.github.com/repos/simonw/datasette/issues/1201 MDEyOklzc3VlQ29tbWVudDc2NjU0NTQ0Mg== simonw 9599 2021-01-25T05:13:59Z 2021-01-25T05:13:59Z OWNER

The big stuff:

  • Database(memory_name=) for shared in-memory databases, closes #1151
  • The _internal database - #1150
  • script type=module support, closes #1186 , #1187
  • Improved design for the .add_database() method 8919f99c2f7f245aca7f94bd53d5ac9d04aa42b5 - which means databases with the same stem can now be opened, #509
  • Adopted Prettier #1166

Smaller:

  • force_https_urls on for publish cloudrun, refs #1178
  • Fixed bug in example nginx config, refs #1091
  • Shrunk ecosystem docs in favour of datasette.io, closes #1182
  • request.full_path property, closes #1184
  • Better PRAGMA error message, closes #1185
  • publish heroku now uses python-3.8.7
  • Plugin testing documentation on using pytest-httpx Closes #1198
  • Contributing docs for Black and Prettier, closes #1167
  • All ?_ parameters now copied to hidden form fields, closes #1194
  • Fixed bug loading database called 'test-database (1).sqlite' - Closes #1181.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Release notes for Datasette 0.54 792904595  
766543387 https://github.com/simonw/datasette/issues/1201#issuecomment-766543387 https://api.github.com/repos/simonw/datasette/issues/1201 MDEyOklzc3VlQ29tbWVudDc2NjU0MzM4Nw== simonw 9599 2021-01-25T05:07:40Z 2021-01-25T05:13:29Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Release notes for Datasette 0.54 792904595  
766536076 https://github.com/simonw/datasette/issues/983#issuecomment-766536076 https://api.github.com/repos/simonw/datasette/issues/983 MDEyOklzc3VlQ29tbWVudDc2NjUzNjA3Ng== simonw 9599 2021-01-25T04:43:53Z 2021-01-25T04:43:53Z OWNER

... actually not going to include this in 0.54, I need to write a couple of plugins myself using it before I even make it available in preview.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
JavaScript plugin hooks mechanism similar to pluggy 712260429  
766535046 https://github.com/simonw/datasette/issues/1195#issuecomment-766535046 https://api.github.com/repos/simonw/datasette/issues/1195 MDEyOklzc3VlQ29tbWVudDc2NjUzNTA0Ng== simonw 9599 2021-01-25T04:40:08Z 2021-01-25T04:40:08Z OWNER

Also: should the view name really be the same for both of these pages?

Where one of them is a canned query and the other is an arbitrary query?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
view_name = "query" for the query page 789336592  
766534748 https://github.com/simonw/datasette/issues/1195#issuecomment-766534748 https://api.github.com/repos/simonw/datasette/issues/1195 MDEyOklzc3VlQ29tbWVudDc2NjUzNDc0OA== simonw 9599 2021-01-25T04:38:56Z 2021-01-25T04:38:56Z OWNER

Here's a diff showing how far I got before I abandoned this particular effort:

diff --git a/datasette/views/base.py b/datasette/views/base.py
index a21b929..04e4aa9 100644
--- a/datasette/views/base.py
+++ b/datasette/views/base.py
@@ -120,7 +120,7 @@ class BaseView:
         handler = getattr(self, request.method.lower(), None)
         return await handler(request, *args, **kwargs)

-    async def render(self, templates, request, context=None):
+    async def render(self, templates, request, context=None, view_name=None):
         context = context or {}
         template = self.ds.jinja_env.select_template(templates)
         template_context = {
@@ -135,7 +135,7 @@ class BaseView:
         }
         return Response.html(
             await self.ds.render_template(
-                template, template_context, request=request, view_name=self.name
+                template, template_context, request=request, view_name=view_name
             )
         )

@@ -155,7 +155,7 @@ class BaseView:


 class DataView(BaseView):
-    name = ""
+    view_name = "no-view-name"
     re_named_parameter = re.compile(":([a-zA-Z0-9_]+)")

     async def options(self, request, *args, **kwargs):
@@ -414,6 +414,10 @@ class DataView(BaseView):
             args["table"] = urllib.parse.unquote_plus(args["table"])
         return _format, args

+    async def get_view_name(self, request, database, hash, **kwargs):
+        # Defaults to self.view_name, but can be over-ridden by subclasses
+        return self.view_name
+
     async def view_get(self, request, database, hash, correct_hash_provided, **kwargs):
         _format, kwargs = await self.get_format(request, database, kwargs)

@@ -424,6 +428,8 @@ class DataView(BaseView):
             # HTML views default to expanding all foreign key labels
             kwargs["default_labels"] = True

+        view_name = await self.get_view_name(request, database, hash, **kwargs)
+
         extra_template_data = {}
         start = time.perf_counter()
         status_code = 200
@@ -489,7 +495,7 @@ class DataView(BaseView):
                 database=database,
                 table=data.get("table"),
                 request=request,
-                view_name=self.name,
+                view_name=view_name,
                 # These will be deprecated in Datasette 1.0:
                 args=request.args,
                 data=data,
@@ -533,7 +539,7 @@ class DataView(BaseView):
                     database=database,
                     table=data.get("table"),
                     request=request,
-                    view_name=self.name,
+                    view_name=view_name,
                 )
                 it_can_render = await await_me_maybe(it_can_render)
                 if it_can_render:
@@ -565,7 +571,7 @@ class DataView(BaseView):
             }
             if "metadata" not in context:
                 context["metadata"] = self.ds.metadata
-            r = await self.render(templates, request=request, context=context)
+            r = await self.render(templates, request=request, context=context, view_name=view_name)
             r.status = status_code

         ttl = request.args.get("_ttl", None)
diff --git a/datasette/views/database.py b/datasette/views/database.py
index f6fd579..e425213 100644
--- a/datasette/views/database.py
+++ b/datasette/views/database.py
@@ -23,7 +23,11 @@ from .base import DatasetteError, DataView


 class DatabaseView(DataView):
-    name = "database"
+    async def get_view_name(self, request, db_name, table_and_format):
+        if request.args.get("sql"):
+            return "query"
+        else:
+            return "database"

     async def data(self, request, database, hash, default_labels=False, _size=None):
         await self.check_permissions(
@@ -145,7 +149,7 @@ class DatabaseView(DataView):


 class DatabaseDownload(DataView):
-    name = "database_download"
+    view_name = "database_download"

     async def view_get(self, request, database, hash, correct_hash_present, **kwargs):
         await self.check_permissions(
diff --git a/datasette/views/index.py b/datasette/views/index.py
index b6b8cbe..d750e3d 100644
--- a/datasette/views/index.py
+++ b/datasette/views/index.py
@@ -16,7 +16,7 @@ COUNT_DB_SIZE_LIMIT = 100 * 1024 * 1024


 class IndexView(BaseView):
-    name = "index"
+    view_name = "index"

     async def get(self, request, as_format):
         await self.check_permission(request, "view-instance")
diff --git a/datasette/views/special.py b/datasette/views/special.py
index 9750dd0..dbd1e00 100644
--- a/datasette/views/special.py
+++ b/datasette/views/special.py
@@ -6,7 +6,7 @@ import secrets


 class JsonDataView(BaseView):
-    name = "json_data"
+    view_name = "json_data"

     def __init__(self, datasette, filename, data_callback, needs_request=False):
         self.ds = datasette
@@ -42,7 +42,7 @@ class JsonDataView(BaseView):


 class PatternPortfolioView(BaseView):
-    name = "patterns"
+    view_name = "patterns"

     async def get(self, request):
         await self.check_permission(request, "view-instance")
@@ -50,7 +50,7 @@ class PatternPortfolioView(BaseView):


 class AuthTokenView(BaseView):
-    name = "auth_token"
+    view_name = "auth_token"

     async def get(self, request):
         token = request.args.get("token") or ""
@@ -68,7 +68,7 @@ class AuthTokenView(BaseView):


 class LogoutView(BaseView):
-    name = "logout"
+    view_name = "logout"

     async def get(self, request):
         if not request.actor:
@@ -87,7 +87,7 @@ class LogoutView(BaseView):


 class PermissionsDebugView(BaseView):
-    name = "permissions_debug"
+    view_name = "permissions_debug"

     async def get(self, request):
         await self.check_permission(request, "view-instance")
@@ -102,7 +102,7 @@ class PermissionsDebugView(BaseView):


 class AllowDebugView(BaseView):
-    name = "allow_debug"
+    view_name = "allow_debug"

     async def get(self, request):
         errors = []
@@ -136,7 +136,7 @@ class AllowDebugView(BaseView):


 class MessagesDebugView(BaseView):
-    name = "messages_debug"
+    view_name = "messages_debug"

     async def get(self, request):
         await self.check_permission(request, "view-instance")
diff --git a/datasette/views/table.py b/datasette/views/table.py
index 0a3504b..45d298a 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -257,7 +257,16 @@ class RowTableShared(DataView):


 class TableView(RowTableShared):
-    name = "table"
+    view_name = "table"
+
+    async def get_view_name(self, request, db_name, table_and_format):
+        canned_query = await self.ds.get_canned_query(
+            db_name, table_and_format, request.actor
+        )
+        if canned_query:
+            return "query"
+        else:
+            return "table"

     async def post(self, request, db_name, table_and_format):
         # Handle POST to a canned query
@@ -923,7 +932,7 @@ async def _sql_params_pks(db, table, pk_values):


 class RowView(RowTableShared):
-    name = "row"
+    view_name = "row"

     async def data(self, request, database, hash, table, pk_path, default_labels=False):
         await self.check_permissions(
diff --git a/tests/test_plugins.py b/tests/test_plugins.py
index 715c7c1..7ce2b1b 100644
--- a/tests/test_plugins.py
+++ b/tests/test_plugins.py
@@ -252,7 +252,7 @@ def test_plugin_config_file(app_client):
             },
         ),
         (
-            "/fixtures/",
+            "/fixtures",
             {
                 "template": "database.html",
                 "database": "fixtures",
@@ -285,6 +285,38 @@ def test_plugin_config_file(app_client):
                 ],
             },
         ),
+        (
+            "/fixtures?sql=select+1+as+one",
+            {
+                "template": "query.html",
+                "database": "fixtures",
+                "table": None,
+                "config": {"depth": "database"},
+                "view_name": "query",
+                "request_path": "/fixtures",
+                "added": 15,
+                "columns": [
+                    "one",
+                ],
+            },
+        ),
+        (
+            "/fixtures/neighborhood_search",
+            {
+                "template": "query.html",
+                "database": "fixtures",
+                "table": None,
+                "config": {"depth": "database"},
+                "view_name": "query",
+                "request_path": "/fixtures/neighborhood_search",
+                "added": 15,
+                "columns": [
+                    "neighborhood",
+                    "name",
+                    "state",
+                ],
+            },
+        ),
     ],
 )
 def test_hook_extra_body_script(app_client, path, expected_extra_body_script):
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
view_name = "query" for the query page 789336592  
766534634 https://github.com/simonw/datasette/issues/1195#issuecomment-766534634 https://api.github.com/repos/simonw/datasette/issues/1195 MDEyOklzc3VlQ29tbWVudDc2NjUzNDYzNA== simonw 9599 2021-01-25T04:38:30Z 2021-01-25T04:38:30Z OWNER

This has proved surprisingly difficult to implement, due to the weird way the QueryView is actually called. The class itself isn't used like other view classes - instead, the .data() methods of both DatabaseView and TableView dispatch out to QueryView.data() when they need to:

https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/table.py#L259-L270

https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/table.py#L290-L294

https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/database.py#L39-L44

It turns out this is a bad pattern because it makes changes like this one WAY harder than they should be.

I think I should clean this up as part of #878.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
view_name = "query" for the query page 789336592  
766525337 https://github.com/simonw/datasette/pull/1204#issuecomment-766525337 https://api.github.com/repos/simonw/datasette/issues/1204 MDEyOklzc3VlQ29tbWVudDc2NjUyNTMzNw== simonw 9599 2021-01-25T04:04:28Z 2021-01-25T04:04:28Z OWNER

Writing the tests will be a bit tricky since we need to confirm that the include_table_top(datasette, database, actor, table) arguments were all passed correctly but the only thing we get back from the plugin is a list of templates. Maybe encode those values into the template names somehow?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
WIP: Plugin includes 793002853  
766524141 https://github.com/simonw/datasette/issues/987#issuecomment-766524141 https://api.github.com/repos/simonw/datasette/issues/987 MDEyOklzc3VlQ29tbWVudDc2NjUyNDE0MQ== simonw 9599 2021-01-25T03:59:55Z 2021-01-25T03:59:55Z OWNER

This is joined with #1191 now, which I've bumped from 0.54.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documented HTML hooks for JavaScript plugin authors 712984738  
766524016 https://github.com/simonw/datasette/issues/1191#issuecomment-766524016 https://api.github.com/repos/simonw/datasette/issues/1191 MDEyOklzc3VlQ29tbWVudDc2NjUyNDAxNg== simonw 9599 2021-01-25T03:59:17Z 2021-01-25T03:59:17Z OWNER

More work can happen in the PR: #1204

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability for plugins to collaborate when adding extra HTML to blocks in default templates 787098345  
766523866 https://github.com/simonw/datasette/issues/1191#issuecomment-766523866 https://api.github.com/repos/simonw/datasette/issues/1191 MDEyOklzc3VlQ29tbWVudDc2NjUyMzg2Ng== simonw 9599 2021-01-25T03:58:34Z 2021-01-25T03:58:34Z OWNER

I've got a good prototype working now, but I'm dropping this from the Datasette 0.54 milestone because it requires a bunch of additional work to make sure it is really well tested and documented.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability for plugins to collaborate when adding extra HTML to blocks in default templates 787098345  
766491911 https://github.com/simonw/datasette/issues/1194#issuecomment-766491911 https://api.github.com/repos/simonw/datasette/issues/1194 MDEyOklzc3VlQ29tbWVudDc2NjQ5MTkxMQ== simonw 9599 2021-01-25T02:02:15Z 2021-01-25T02:02:15Z OWNER

I'm going to copy across anything starting with an underscore.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_size= argument is not persisted by hidden form fields in the table filters 788447787  
766491566 https://github.com/simonw/datasette/issues/1167#issuecomment-766491566 https://api.github.com/repos/simonw/datasette/issues/1167 MDEyOklzc3VlQ29tbWVudDc2NjQ5MTU2Ng== simonw 9599 2021-01-25T02:01:19Z 2021-01-25T02:01:19Z OWNER

New documentation section here (I documented Black as well): https://docs.datasette.io/en/latest/contributing.html#code-formatting

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add Prettier to contributing documentation 777145954  
766487520 https://github.com/simonw/datasette/issues/1167#issuecomment-766487520 https://api.github.com/repos/simonw/datasette/issues/1167 MDEyOklzc3VlQ29tbWVudDc2NjQ4NzUyMA== simonw 9599 2021-01-25T01:44:43Z 2021-01-25T01:44:43Z OWNER

Thanks @benpickles, I just merged that in. I'll use it in the documentation.

# To check code is conformant
npm run prettier -- --check
# To fix it if it isn't
npm run fix
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add Prettier to contributing documentation 777145954  
766484915 https://github.com/simonw/datasette/issues/983#issuecomment-766484915 https://api.github.com/repos/simonw/datasette/issues/983 MDEyOklzc3VlQ29tbWVudDc2NjQ4NDkxNQ== simonw 9599 2021-01-25T01:33:29Z 2021-01-25T01:33:29Z OWNER

I'm going to ship a version of this in Datasette 0.54 with a warning that the interface should be considered unstable (see #1202) so that we can start trying this out.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
JavaScript plugin hooks mechanism similar to pluggy 712260429  
766484435 https://github.com/simonw/datasette/issues/1179#issuecomment-766484435 https://api.github.com/repos/simonw/datasette/issues/1179 MDEyOklzc3VlQ29tbWVudDc2NjQ4NDQzNQ== simonw 9599 2021-01-25T01:31:36Z 2021-01-25T01:31:36Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make original path available to render hooks 780278550  
766484257 https://github.com/simonw/datasette/issues/1179#issuecomment-766484257 https://api.github.com/repos/simonw/datasette/issues/1179 MDEyOklzc3VlQ29tbWVudDc2NjQ4NDI1Nw== simonw 9599 2021-01-25T01:30:57Z 2021-01-25T01:30:57Z OWNER

The challenge here is figuring out what the original path, without the .format, actually was - while taking into account that Datasette has a special case for tables that themselves end in a .something.

The path_with_format() function nearly does what we need here:

https://github.com/simonw/datasette/blob/b6a7b58fa01af0cd5a5e94bd17d686d283a46819/datasette/utils/__init__.py#L710-L729

It can be called with replace_format="csv" to REMOVE the .csv format and replace it with something else.

Problem is, we want to use it to get rid of the format entirely.

We could update path_with_format() to accept format='' to mean "remove the format entirely", but it's a bit messy. It may be better to reconsider the design of path_with_format() and related utility functions entirely.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make original path available to render hooks 780278550  
766466030 https://github.com/simonw/datasette/issues/1191#issuecomment-766466030 https://api.github.com/repos/simonw/datasette/issues/1191 MDEyOklzc3VlQ29tbWVudDc2NjQ2NjAzMA== simonw 9599 2021-01-25T00:11:04Z 2021-01-25T00:11:04Z OWNER

I can combine this with #987 - each of these areas of the page can be wrapped in a <div> with a class that matches the name of the plugin hook, that way JavaScript plugins can append their content in the same place as Python plugins.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability for plugins to collaborate when adding extra HTML to blocks in default templates 787098345  
766465719 https://github.com/simonw/datasette/issues/1154#issuecomment-766465719 https://api.github.com/repos/simonw/datasette/issues/1154 MDEyOklzc3VlQ29tbWVudDc2NjQ2NTcxOQ== simonw 9599 2021-01-25T00:09:22Z 2021-01-25T00:09:22Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation for new _internal database and tables 771208009  
766464136 https://github.com/simonw/datasette/issues/1202#issuecomment-766464136 https://api.github.com/repos/simonw/datasette/issues/1202 MDEyOklzc3VlQ29tbWVudDc2NjQ2NDEzNg== simonw 9599 2021-01-25T00:01:02Z 2021-01-25T00:01:02Z OWNER

I'm going to use the existing .. warning:: pattern for this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation convention for marking unstable APIs. 792931244  
766463496 https://github.com/simonw/datasette/issues/1090#issuecomment-766463496 https://api.github.com/repos/simonw/datasette/issues/1090 MDEyOklzc3VlQ29tbWVudDc2NjQ2MzQ5Ng== simonw 9599 2021-01-24T23:57:00Z 2021-01-24T23:57:00Z OWNER

Related: I built datasette-leaflet-freedraw which turns any canned query field called freedraw or something_freedraw into an interactive map that you can draw on to create a GeoJSON MultiPolygon.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Custom widgets for canned query forms 741862364  
766462475 https://github.com/simonw/datasette/issues/1202#issuecomment-766462475 https://api.github.com/repos/simonw/datasette/issues/1202 MDEyOklzc3VlQ29tbWVudDc2NjQ2MjQ3NQ== simonw 9599 2021-01-24T23:49:28Z 2021-01-24T23:50:33Z OWNER

Can use an "admonition" similar to this:

.. warning::
    Restricting access to tables and views in this way will NOT prevent users from querying them using arbitrary SQL queries, `like this <https://latest.datasette.io/fixtures?sql=select+*+from+facetable>`__ for example.

As seen on https://docs.datasette.io/en/stable/authentication.html#controlling-access-to-specific-tables-and-views

Documentation: https://docutils.sourceforge.io/docs/ref/rst/directives.html#specific-admonitions

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation convention for marking unstable APIs. 792931244  
766462197 https://github.com/simonw/datasette/issues/1154#issuecomment-766462197 https://api.github.com/repos/simonw/datasette/issues/1154 MDEyOklzc3VlQ29tbWVudDc2NjQ2MjE5Nw== simonw 9599 2021-01-24T23:47:06Z 2021-01-24T23:47:06Z OWNER

I'm going to document this but mark it as unstable, using a new documentation convention for marking unstable APIs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation for new _internal database and tables 771208009  
766434629 https://github.com/simonw/datasette/issues/1179#issuecomment-766434629 https://api.github.com/repos/simonw/datasette/issues/1179 MDEyOklzc3VlQ29tbWVudDc2NjQzNDYyOQ== simonw 9599 2021-01-24T21:23:47Z 2021-01-24T21:23:47Z OWNER

I'm just going to do path and full_path (which includes the querystring). Thedatasette.absolute_url()` method can be used by plugins that need the full URL.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make original path available to render hooks 780278550  
766433153 https://github.com/simonw/datasette/issues/1190#issuecomment-766433153 https://api.github.com/repos/simonw/datasette/issues/1190 MDEyOklzc3VlQ29tbWVudDc2NjQzMzE1Mw== simonw 9599 2021-01-24T21:13:25Z 2021-01-24T21:13:25Z OWNER

This ties in to a bunch of other ideas that are in flight at the moment.

If you're publishing databases by uploading them, how do you attach metadata? Ideally by baking it into the database file itself, using the mechanism from #1169.

How could this interact with the datasette insert concept from #1163? Could you pass a CSV file to the upload command and have that converted and uploaded for you, or would you create the database file locally using datasette insert and then upload it as a separate datasette upload step?

Lots to think about here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
`datasette publish upload` mechanism for uploading databases to an existing Datasette instance 787098146  
766430644 https://github.com/simonw/datasette/issues/1190#issuecomment-766430644 https://api.github.com/repos/simonw/datasette/issues/1190 MDEyOklzc3VlQ29tbWVudDc2NjQzMDY0NA== simonw 9599 2021-01-24T20:57:03Z 2021-01-24T20:57:03Z OWNER

I really like this idea. It feels like an opportunity for a plugin that adds two things: an API endpoint to Datasette for accepting uploaded databases, and a datasette publish upload subcommand which can upload files to that endpoint (with some kind of authentication mechanism).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
`datasette publish upload` mechanism for uploading databases to an existing Datasette instance 787098146  
766430111 https://github.com/simonw/datasette/issues/1197#issuecomment-766430111 https://api.github.com/repos/simonw/datasette/issues/1197 MDEyOklzc3VlQ29tbWVudDc2NjQzMDExMQ== simonw 9599 2021-01-24T20:53:40Z 2021-01-24T20:53:40Z OWNER

https://devcenter.heroku.com/articles/slug-compiler#slug-size says that the maximum allowed size is 500MB - my hunch is that the Datasette application itself weighs in at only a dozen or so MB but I haven't measured it. So I would imagine anything up to around 450MB should work OK on Heroku.

Cloud Run works for up to about 2GB in my experience.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
DB size limit for publishing with Heroku 791381623  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);