issue_comments

3,377 rows sorted by updated_at descending

View and edit SQL

Suggested facets: reactions, created_at (date), updated_at (date)

issue

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
667585598 https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667585598 https://api.github.com/repos/simonw/sqlite-utils/issues/130 MDEyOklzc3VlQ29tbWVudDY2NzU4NTU5OA== simonw 9599 2020-08-01T20:51:28Z 2020-08-01T20:51:28Z OWNER

CLI documentation: https://github.com/simonw/sqlite-utils/commit/57e4eb8e5564af5d97f892b3be8342451ee177a2?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support tokenize option for FTS 671130371  
667585561 https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667585561 https://api.github.com/repos/simonw/sqlite-utils/issues/130 MDEyOklzc3VlQ29tbWVudDY2NzU4NTU2MQ== simonw 9599 2020-08-01T20:50:59Z 2020-08-01T20:50:59Z OWNER

Turns out it works for FTS4 as well: https://www.sqlite.org/fts3.html#tokenizer

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support tokenize option for FTS 671130371  
667584567 https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667584567 https://api.github.com/repos/simonw/sqlite-utils/issues/130 MDEyOklzc3VlQ29tbWVudDY2NzU4NDU2Nw== simonw 9599 2020-08-01T20:41:09Z 2020-08-01T20:41:09Z OWNER

API documentation here: https://github.com/simonw/sqlite-utils/commit/617e6f070c85be66ea04c80b78dafd08c875f8c8?short_path=e156262#diff-e1562629b8def6da772d9b0903faf703

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support tokenize option for FTS 671130371  
667431123 https://github.com/simonw/datasette/issues/900#issuecomment-667431123 https://api.github.com/repos/simonw/datasette/issues/900 MDEyOklzc3VlQ29tbWVudDY2NzQzMTEyMw== simonw 9599 2020-07-31T23:56:33Z 2020-07-31T23:56:33Z OWNER

I think this is the same issue as #865. I'll look at these together!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Some links don't honor base_url 661605489  
667430790 https://github.com/simonw/datasette/issues/899#issuecomment-667430790 https://api.github.com/repos/simonw/datasette/issues/899 MDEyOklzc3VlQ29tbWVudDY2NzQzMDc5MA== simonw 9599 2020-07-31T23:54:40Z 2020-07-31T23:54:40Z OWNER

There's no mechanism that can do this at the moment.

You could absolutely support this with a plugin, probably using the asgi_wrapper plugin hook. There's an existing package at https://pypi.org/project/asgi-ratelimit/ which may be usable for this - it may even be possible to configure that using https://github.com/simonw/datasette-configure-asgi rather than using it to write a custom plugin.

Using a separate revers proxy would also be a good way to solve this. It depends which option would work best in your environment.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
How to setup a request limit per user 660827546  
667430352 https://github.com/simonw/datasette/issues/913#issuecomment-667430352 https://api.github.com/repos/simonw/datasette/issues/913 MDEyOklzc3VlQ29tbWVudDY2NzQzMDM1Mg== simonw 9599 2020-07-31T23:52:10Z 2020-07-31T23:52:10Z OWNER

The bigger question here is when this mechanism should be used in place of metadata.json or metadata.yml. Especially since I'm already considering renaming or reworking that mechanism since plugin configuration has nothing to do with database metadata: #493

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for passing additional options to `datasette my.db` that affect plugins 670209331  
667429616 https://github.com/simonw/datasette/issues/913#issuecomment-667429616 https://api.github.com/repos/simonw/datasette/issues/913 MDEyOklzc3VlQ29tbWVudDY2NzQyOTYxNg== simonw 9599 2020-07-31T23:48:25Z 2020-07-31T23:49:59Z OWNER

I could let plugins add additional options to datasette serve - but what if two plugins both try to register an option with the same name?

A better solution could be to use the existing --config option - and allow plugins to register their own, namespaced config options. So you could do things like:

datasette my.db --config datasette-insert:unsafe:1

Maybe even drop the datasette- prefix?

datasette my.db --config insert:unsafe:1

I think I prefer keeping the prefix to be honest - it makes it more obvious that this is a setting which comes from a specific named plugin.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for passing additional options to `datasette my.db` that affect plugins 670209331  
667429690 https://github.com/simonw/datasette/issues/913#issuecomment-667429690 https://api.github.com/repos/simonw/datasette/issues/913 MDEyOklzc3VlQ29tbWVudDY2NzQyOTY5MA== simonw 9599 2020-07-31T23:48:48Z 2020-07-31T23:48:48Z OWNER

Here's the code in Datasette that parses --config options at the moment:

https://github.com/simonw/datasette/blob/7ca8c0521ac1ea48a3cd8d0fe9275d1316e54b43/datasette/cli.py#L25-L40

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for passing additional options to `datasette my.db` that affect plugins 670209331  
667424128 https://github.com/simonw/datasette/issues/849#issuecomment-667424128 https://api.github.com/repos/simonw/datasette/issues/849 MDEyOklzc3VlQ29tbWVudDY2NzQyNDEyOA== simonw 9599 2020-07-31T23:21:56Z 2020-07-31T23:23:24Z OWNER

I'm going to change the default branch on the GitHub repository. If something breaks I can always change it back again.

Done that! Default is now main.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename master branch to main 639072811  
667424020 https://github.com/simonw/datasette/issues/849#issuecomment-667424020 https://api.github.com/repos/simonw/datasette/issues/849 MDEyOklzc3VlQ29tbWVudDY2NzQyNDAyMA== simonw 9599 2020-07-31T23:21:30Z 2020-07-31T23:21:30Z OWNER

https://github.com/simonw/datasette/tree/main branch now exists and will automatically mirror master (and vice-versa).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename master branch to main 639072811  
667295759 https://github.com/simonw/datasette/issues/849#issuecomment-667295759 https://api.github.com/repos/simonw/datasette/issues/849 MDEyOklzc3VlQ29tbWVudDY2NzI5NTc1OQ== simonw 9599 2020-07-31T18:45:35Z 2020-07-31T18:45:35Z OWNER

Watch out for places in the documentation that might link to master - e.g. here:

https://github.com/simonw/datasette/blob/2d7fa8b9058dfbf9c7c371cdeec115d32a177dc9/docs/custom_templates.rst#L247

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename master branch to main 639072811  
664105302 https://github.com/simonw/sqlite-utils/issues/124#issuecomment-664105302 https://api.github.com/repos/simonw/sqlite-utils/issues/124 MDEyOklzc3VlQ29tbWVudDY2NDEwNTMwMg== simonw 9599 2020-07-27T03:54:24Z 2020-07-30T22:57:51Z OWNER

Documentation: https://github.com/simonw/sqlite-utils/commit/814d4a7f90991be865d38aac45ff12e36df1c67d?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40

You can pass named parameters to the query using -p:

$ sqlite-utils query dogs.db "select :num * :num2" -p num 5 -p num2 6
[{":num * :num2": 30}]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
sqlite-utils query should support named parameters 665802405  
666752039 https://github.com/simonw/sqlite-utils/issues/129#issuecomment-666752039 https://api.github.com/repos/simonw/sqlite-utils/issues/129 MDEyOklzc3VlQ29tbWVudDY2Njc1MjAzOQ== simonw 9599 2020-07-30T22:40:55Z 2020-07-30T22:40:55Z OWNER

This should be a separate command from insert-files. SQLite Archives should use a table with this schema:

CREATE TABLE sqlar(
  name TEXT PRIMARY KEY,  -- name of the file
  mode INT,               -- access permissions
  mtime INT,              -- last modification time
  sz INT,                 -- original file size
  data BLOB               -- compressed content
);

insert-files currently treats the table name as a required argument - but it's not necessary for this table. Also there shouldn't be any support for the --column option.

So if I write this command it should be this instead:

sqlite-utils sqlar files.db file.txt file2.txt

But at that point, why bother? Users can use sqlite3 files.db -Ac *.txt instead.

So I'm not going to bother implementing this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"insert-files --sqlar" for creating SQLite archives 668308777  
666063689 https://github.com/simonw/sqlite-utils/issues/127#issuecomment-666063689 https://api.github.com/repos/simonw/sqlite-utils/issues/127 MDEyOklzc3VlQ29tbWVudDY2NjA2MzY4OQ== simonw 9599 2020-07-30T03:08:51Z 2020-07-30T03:08:51Z OWNER

Documentation at the bottom of this section: https://github.com/simonw/sqlite-utils/blob/8fe1e6d1be021aeeb8f08b0f77f03b75a83b6f75/docs/cli.rst#inserting-binary-data-from-files

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to insert files piped to insert-files stdin 666040390  
666047928 https://github.com/simonw/sqlite-utils/issues/127#issuecomment-666047928 https://api.github.com/repos/simonw/sqlite-utils/issues/127 MDEyOklzc3VlQ29tbWVudDY2NjA0NzkyOA== simonw 9599 2020-07-30T02:31:05Z 2020-07-30T02:31:05Z OWNER

Maybe could do this using an improved version of this lambda? Could teach it to look for - and read from sys.stdin if it sees it.

https://github.com/simonw/sqlite-utils/blob/710454d72aed5094573e642344fd075a0ef5372c/sqlite_utils/cli.py#L839

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to insert files piped to insert-files stdin 666040390  
666046819 https://github.com/simonw/sqlite-utils/issues/129#issuecomment-666046819 https://api.github.com/repos/simonw/sqlite-utils/issues/129 MDEyOklzc3VlQ29tbWVudDY2NjA0NjgxOQ== simonw 9599 2020-07-30T02:28:34Z 2020-07-30T02:28:34Z OWNER

This code looks useful as inspiration: https://github.com/j4mie/sqlsite/blob/f2dadb8db5ed7880f8872b6591d8cb1487f777ea/sqlsite/sqlar.py

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"insert-files --sqlar" for creating SQLite archives 668308777  
666010395 https://github.com/simonw/datasette/issues/909#issuecomment-666010395 https://api.github.com/repos/simonw/datasette/issues/909 MDEyOklzc3VlQ29tbWVudDY2NjAxMDM5NQ== simonw 9599 2020-07-30T00:56:17Z 2020-07-30T00:56:17Z OWNER
$ curl -I https://latest.datasette.io/fixtures.db
HTTP/1.1 200 OK
content-disposition: attachment; filename="fixtures.db"
content-type: application/octet-stream
Date: Thu, 30 Jul 2020 00:56:05 GMT
Server: Google Frontend
Transfer-Encoding: chunked
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
AsgiFileDownload: filename not correctly passed 667467128  
665854704 https://github.com/simonw/datasette/issues/909#issuecomment-665854704 https://api.github.com/repos/simonw/datasette/issues/909 MDEyOklzc3VlQ29tbWVudDY2NTg1NDcwNA== simonw 9599 2020-07-29T19:22:31Z 2020-07-29T19:22:31Z OWNER

I think this results in a bug where the "download database" link doesn't include the correct filename: https://github.com/simonw/datasette/blob/549b1c2063db48c4622ee5c7b478a1e3cbc1ac07/datasette/views/database.py#L110-L131

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
AsgiFileDownload: filename not correctly passed 667467128  
665663131 https://github.com/simonw/datasette/pull/910#issuecomment-665663131 https://api.github.com/repos/simonw/datasette/issues/910 MDEyOklzc3VlQ29tbWVudDY2NTY2MzEzMQ== codecov[bot] 22429695 2020-07-29T13:26:14Z 2020-07-29T13:26:14Z NONE

Codecov Report

Merging #910 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #910   +/-   ##
=======================================
  Coverage   83.55%   83.55%           
=======================================
  Files          27       27           
  Lines        3666     3666           
=======================================
  Hits         3063     3063           
  Misses        603      603           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c33b42...3493915. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Update pytest requirement from <5.5.0,>=5.2.2 to >=5.2.2,<6.1.0 667840539  
664683608 https://github.com/simonw/sqlite-utils/issues/128#issuecomment-664683608 https://api.github.com/repos/simonw/sqlite-utils/issues/128 MDEyOklzc3VlQ29tbWVudDY2NDY4MzYwOA== simonw 9599 2020-07-27T23:09:22Z 2020-07-27T23:09:22Z OWNER

This seems to work, but needs more tests:
```diff
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index d6b9ecf..ee26433 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -7,6 +7,7 @@ import itertools
import json
import os
import pathlib
+import uuid

SQLITE_MAX_VARS = 999

@@ -40,11 +41,13 @@ COLUMN_TYPE_MAPPING = {
str: "TEXT",
bytes.class: "BLOB",
bytes: "BLOB",
+ memoryview: "BLOB",
datetime.datetime: "TEXT",
datetime.date: "TEXT",
datetime.time: "TEXT",
decimal.Decimal: "FLOAT",
None.class: "TEXT",
+ uuid.UUID: "TEXT",
# SQLite explicit types
"TEXT": "TEXT",
"INTEGER": "INTEGER",
@@ -1336,6 +1339,8 @@ def jsonify_if_needed(value):
return json.dumps(value, default=repr)
elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)):
return value.isoformat()
+ elif isinstance(value, uuid.UUID):
+ return str(value)
else:
return value
```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support UUID and memoryview types 666639051  
664163524 https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664163524 https://api.github.com/repos/simonw/sqlite-utils/issues/122 MDEyOklzc3VlQ29tbWVudDY2NDE2MzUyNA== simonw 9599 2020-07-27T07:10:41Z 2020-07-27T07:10:41Z OWNER

Docs: https://github.com/simonw/sqlite-utils/blob/ebc802f7ff0e640b6ae11ea525290fea0115228c/docs/cli.rst#inserting-binary-data-from-files

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CLI utility for inserting binary files into SQLite 665700495  
664163206 https://github.com/simonw/sqlite-utils/issues/127#issuecomment-664163206 https://api.github.com/repos/simonw/sqlite-utils/issues/127 MDEyOklzc3VlQ29tbWVudDY2NDE2MzIwNg== simonw 9599 2020-07-27T07:10:05Z 2020-07-27T07:10:05Z OWNER

I tried to get this working but it was a bit tricky because - doesn't behave like a regular pathlib.Path - needs a bit more thought on how the implementation would work.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to insert files piped to insert-files stdin 666040390  
664128071 https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071 https://api.github.com/repos/simonw/sqlite-utils/issues/122 MDEyOklzc3VlQ29tbWVudDY2NDEyODA3MQ== simonw 9599 2020-07-27T05:30:54Z 2020-07-27T05:30:54Z OWNER

Inserting files by piping them in should work - but since a filename cannot be derived this will need a --name blah.gif option.

cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CLI utility for inserting binary files into SQLite 665700495  
664127741 https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664127741 https://api.github.com/repos/simonw/sqlite-utils/issues/122 MDEyOklzc3VlQ29tbWVudDY2NDEyNzc0MQ== simonw 9599 2020-07-27T05:29:48Z 2020-07-27T05:29:48Z OWNER

Test command:

sqlite-utils insert-files gifs.db *.gif \
    -c filename:filename \
    -c filepath:filepath \
    -c absolutepath:absolutepath \
    -c sha256:sha256 \
    -c md5:md5 \
    -c content:content \
    -c mtime:mtime \
    -c ctime:ctime \
    -c mtime_iso:mtime_iso \
    -c ctime_iso:ctime_iso \
    -c size:size \
    --pk absolutepath
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CLI utility for inserting binary files into SQLite 665700495  
663931279 https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931279 https://api.github.com/repos/simonw/sqlite-utils/issues/122 MDEyOklzc3VlQ29tbWVudDY2MzkzMTI3OQ== simonw 9599 2020-07-26T03:33:23Z 2020-07-27T04:30:49Z OWNER

One idea: sqlite-utils insert-files

It could work something like this:

sqlite-utils insert-files files.db /tmp/blah.jpg /tmp/foo.gif \
  --table files \
  -c key:filename -c hash:sha256 -c body:content \
  --pk key

This would insert those two image files into the database in a table called files with a schema that looks something like this:

CREATE TABLE files (
    key text primary key,
    hash text,
    body blob
);

The -c key:filename options here are the most interesting: they let you create the table with a specific layout. The bit before the : is the column name. The bit after the : can be a range of different things:

  • filename - just the filename
  • filepath - the full filepath (provided on the command-line)
  • absolutepath - the filepath expanded to start with /home/... or whatever
  • sha256 - the SHA256 of the contents
  • md5 - the MD5
  • content - the binary content itself
  • mtime - the mtime (floating point timestamp)
  • ctime - the ctime (floating point timestamp)
  • mtime_iso - the mtime as an ISO datetime
  • ctime_iso - the mtime as an ISO datetime
  • size - the size of the file in bytes
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CLI utility for inserting binary files into SQLite 665700495  
664106621 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-664106621 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY2NDEwNjYyMQ== simonw 9599 2020-07-27T04:01:13Z 2020-07-27T04:01:13Z OWNER

Work in progress in transform branch here: https://github.com/simonw/sqlite-utils/tree/transform

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740  
664106405 https://github.com/simonw/sqlite-utils/issues/126#issuecomment-664106405 https://api.github.com/repos/simonw/sqlite-utils/issues/126 MDEyOklzc3VlQ29tbWVudDY2NDEwNjQwNQ== simonw 9599 2020-07-27T04:00:08Z 2020-07-27T04:00:33Z OWNER
$ echo '[
    {
        "name": "transparent.gif",
        "content": {
            "$base64": true,
            "encoded": "R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"
        }
    }
]' | sqlite-utils insert trans.db files - --pk=name
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to insert binary data on the CLI using JSON 665819048  
664065597 https://github.com/simonw/sqlite-utils/issues/126#issuecomment-664065597 https://api.github.com/repos/simonw/sqlite-utils/issues/126 MDEyOklzc3VlQ29tbWVudDY2NDA2NTU5Nw== simonw 9599 2020-07-27T00:51:11Z 2020-07-27T00:51:11Z OWNER

I'm going to implement this as the reverse of #125 - binary columns in JSON are now output like this:

  {
    "name": "lorem.txt",
    "mode": 33188,
    "mtime": 1595805965,
    "sz": 16984,
    "data": {
      "$base64": true,
      "encoded": "eJzt0c1xAyEMBeC7q1ABHleR3HxNAQrIjmb4M0gelx+RTY7p4N2WBYT0vmufUknH8kq5lz5pqRFXsTOl3pYkE/NJnHXoStruJEVjc0mOCyTqq/ZMJnXEZW1Js2ZvRm5U+DPKk9hRWqjyvTFx0YfzhT6MpGmN2lR1fzxjyfVMD9dFrS+bnkleMpMam/ZGXgrX1I/K+5Au3S/9lNQRh0k4Gq/RUz8GiKfsQm+7JLsJ6fTo5JhVG00ZU76kZZkxePx49uIjnpNoJyYlWUsoaSl/CcVATje/Kxu13RANnrHweaH3V5Jh4jvGyKCnxJLiXPKhmW3fiCnG7Jql7RR3UvFo8jJ4z039dtOkTFmWzL1be9lt8A5II471m6vXy+l0BR/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984PuP7xubBoN9"
    }
  }
]

So the sqlite-utils insert command should learn to spot {"$base64": true...} values and base64 decode them before inserting them.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to insert binary data on the CLI using JSON 665819048  
664065341 https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664065341 https://api.github.com/repos/simonw/sqlite-utils/issues/125 MDEyOklzc3VlQ29tbWVudDY2NDA2NTM0MQ== simonw 9599 2020-07-27T00:49:41Z 2020-07-27T00:49:41Z OWNER

Documentation: https://github.com/simonw/sqlite-utils/commit/20e543e9a492f2e764caae73c38e87f18eaec444?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Output binary columns in "sqlite-utils query" JSON 665817570  
664062546 https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664062546 https://api.github.com/repos/simonw/sqlite-utils/issues/125 MDEyOklzc3VlQ29tbWVudDY2NDA2MjU0Ng== simonw 9599 2020-07-27T00:33:03Z 2020-07-27T00:33:03Z OWNER

I'm going to imitate how Datasette solves this problem:

[
  {
    "name": "lorem.txt",
    "mode": 33188,
    "mtime": 1595805965,
    "sz": 16984,
    "data": {
      "$base64": true,
      "encoded": "eJzt0c1xAyEMBeC7q1ABHleR3HxNAQrIjmb4M0gelx+RTY7p4N2WBYT0vmufUknH8kq5lz5pqRFXsTOl3pYkE/NJnHXoStruJEVjc0mOCyTqq/ZMJnXEZW1Js2ZvRm5U+DPKk9hRWqjyvTFx0YfzhT6MpGmN2lR1fzxjyfVMD9dFrS+bnkleMpMam/ZGXgrX1I/K+5Au3S/9lNQRh0k4Gq/RUz8GiKfsQm+7JLsJ6fTo5JhVG00ZU76kZZkxePx49uIjnpNoJyYlWUsoaSl/CcVATje/Kxu13RANnrHweaH3V5Jh4jvGyKCnxJLiXPKhmW3fiCnG7Jql7RR3UvFo8jJ4z039dtOkTFmWzL1be9lt8A5II471m6vXy+l0BR/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984PuP7xubBoN9"
    }
  }
]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Output binary columns in "sqlite-utils query" JSON 665817570  
664048720 https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664048720 https://api.github.com/repos/simonw/sqlite-utils/issues/122 MDEyOklzc3VlQ29tbWVudDY2NDA0ODcyMA== simonw 9599 2020-07-26T22:32:50Z 2020-07-26T22:33:20Z OWNER

This seems to work in creating a SQLite archive containing all .gif files in the current directory:

/usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 archive.db -A -c *.gif

Then listing files like this:

$ /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 archive.db -A -t
copyable.gif
debug-allow.gif
flash.gif
table-md.gif

Here's the schema:

$ sqlite3 archive.db .schema                                  
CREATE TABLE sqlar(
  name TEXT PRIMARY KEY,  -- name of the file
  mode INT,               -- access permissions
  mtime INT,              -- last modification time
  sz INT,                 -- original file size
  data BLOB               -- compressed content
);
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CLI utility for inserting binary files into SQLite 665700495  
664048432 https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664048432 https://api.github.com/repos/simonw/sqlite-utils/issues/122 MDEyOklzc3VlQ29tbWVudDY2NDA0ODQzMg== simonw 9599 2020-07-26T22:29:31Z 2020-07-26T22:29:31Z OWNER

I'm trying to play with sqlite3 -A on my Mac.

sqlite3 -A tells me that it's an unknown option - but I used brew info sqlite to find my homebrew installed version and it turns out this works:

 % /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 -A
Wrong number of arguments.  Usage:
.archive ...             Manage SQL archives
   Each command must have exactly one of the following options:
     -c, --create               Create a new archive
     -u, --update               Add or update files with changed mtime
     -i, --insert               Like -u but always add even if unchanged
     -t, --list                 List contents of archive
     -x, --extract              Extract files from archive
   Optional arguments:
     -v, --verbose              Print each filename as it is processed
     -f FILE, --file FILE       Use archive FILE (default is current db)
     -a FILE, --append FILE     Open FILE using the apndvfs VFS
     -C DIR, --directory DIR    Read/extract files from directory DIR
     -n, --dryrun               Show the SQL that would have occurred
   Examples:
     .ar -cf ARCHIVE foo bar  # Create ARCHIVE from files foo and bar
     .ar -tf ARCHIVE          # List members of ARCHIVE
     .ar -xvf ARCHIVE         # Verbosely extract files from ARCHIVE
   See also:
      http://sqlite.org/cli.html#sqlar_archive_support
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CLI utility for inserting binary files into SQLite 665700495  
664013338 https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664013338 https://api.github.com/repos/simonw/sqlite-utils/issues/122 MDEyOklzc3VlQ29tbWVudDY2NDAxMzMzOA== simonw 9599 2020-07-26T16:57:35Z 2020-07-26T16:57:35Z OWNER

I should consider easy compatibility with https://www.sqlite.org/sqlar.html

An SQLite Archive is an ordinary SQLite database file that contains the following table as part of its schema:
CREATE TABLE sqlar( name TEXT PRIMARY KEY, -- name of the file mode INT, -- access permissions mtime INT, -- last modification time sz INT, -- original file size data BLOB -- compressed content );
Each row of the SQLAR table holds the content of a single file. The filename (the full pathname relative to the root of the archive) is in the "name" field. The "mode" field is an integer which is the unix-style access permissions for the file. "mtime" is the modification time of the file in seconds since 1970. "sz" is the original uncompressed size of the file. The "data" field contains the file content. The content is usually compressed using Deflate, though not always. If the "sz" field is equal to the size of the "data" field, then the content is stored uncompressed.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CLI utility for inserting binary files into SQLite 665700495  
664012247 https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247 https://api.github.com/repos/simonw/sqlite-utils/issues/125 MDEyOklzc3VlQ29tbWVudDY2NDAxMjI0Nw== simonw 9599 2020-07-26T16:48:46Z 2020-07-26T16:48:46Z OWNER

I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says "these columns are base64 encoded, store the decoded data in a BLOB".

That would solve inserting binary data using JSON too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Output binary columns in "sqlite-utils query" JSON 665817570  
664012148 https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012148 https://api.github.com/repos/simonw/sqlite-utils/issues/125 MDEyOklzc3VlQ29tbWVudDY2NDAxMjE0OA== simonw 9599 2020-07-26T16:47:51Z 2020-07-26T16:47:51Z OWNER

Best solution I can think of is to return the data as base64. It's a bit nasty since it means you can't round trip it back again.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Output binary columns in "sqlite-utils query" JSON 665817570  
663931426 https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931426 https://api.github.com/repos/simonw/sqlite-utils/issues/122 MDEyOklzc3VlQ29tbWVudDY2MzkzMTQyNg== simonw 9599 2020-07-26T03:35:58Z 2020-07-26T16:44:33Z OWNER

Related: #123 (--raw option)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CLI utility for inserting binary files into SQLite 665700495  
663931662 https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931662 https://api.github.com/repos/simonw/sqlite-utils/issues/122 MDEyOklzc3VlQ29tbWVudDY2MzkzMTY2Mg== simonw 9599 2020-07-26T03:40:29Z 2020-07-26T03:40:29Z OWNER

Maybe support --replace for replacing images with an existing primary key.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CLI utility for inserting binary files into SQLite 665700495  
663931317 https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931317 https://api.github.com/repos/simonw/sqlite-utils/issues/122 MDEyOklzc3VlQ29tbWVudDY2MzkzMTMxNw== simonw 9599 2020-07-26T03:33:54Z 2020-07-26T03:33:54Z OWNER

The command also accepts one or more directories, in which case it will recursively scan them for all files that they contain.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CLI utility for inserting binary files into SQLite 665700495  
663924220 https://github.com/simonw/datasette/pull/868#issuecomment-663924220 https://api.github.com/repos/simonw/datasette/issues/868 MDEyOklzc3VlQ29tbWVudDY2MzkyNDIyMA== joshmgrant 702729 2020-07-26T01:29:00Z 2020-07-26T01:29:00Z NONE

Ok, so it's been a while but I think I'm making progress. The good news: I have come up with a configuration change for running the tests on Windows on GitHub Actions. The bad news: there's a bunch of extraneous errors on the Windows case. I think this is due to Windows file IO and sqlite in a lot of cases, so I'm working through it. I will eventually clean up this PR.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
initial windows ci setup 646448486  
650340914 https://github.com/simonw/datasette/pull/868#issuecomment-650340914 https://api.github.com/repos/simonw/datasette/issues/868 MDEyOklzc3VlQ29tbWVudDY1MDM0MDkxNA== codecov[bot] 22429695 2020-06-26T18:53:02Z 2020-07-26T01:21:08Z NONE

Codecov Report

Merging #868 into master will increase coverage by 0.09%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #868      +/-   ##
==========================================
+ Coverage   83.31%   83.40%   +0.09%     
==========================================
  Files          27       27              
  Lines        3595     3634      +39     
==========================================
+ Hits         2995     3031      +36     
- Misses        600      603       +3     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/app.py</td> <td>95.99% <0.00%> (-0.29%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/views/database.py</td> <td>96.37% <0.00%> (-0.08%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/views/base.py</td> <td>93.39% <0.00%> (-0.05%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/views/table.py</td> <td>95.67% <0.00%> (-0.03%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/hookspecs.py</td> <td>100.00% <0.00%> (ø)</td> <td></td> </tr> <tr> <td>datasette/utils/__init__.py</td> <td>93.93% <0.00%> (+0.05%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/views/special.py</td> <td>81.17% <0.00%> (+3.39%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8a5f81...da21299. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
initial windows ci setup 646448486  
663779460 https://github.com/simonw/datasette/issues/906#issuecomment-663779460 https://api.github.com/repos/simonw/datasette/issues/906 MDEyOklzc3VlQ29tbWVudDY2Mzc3OTQ2MA== simonw 9599 2020-07-25T00:07:10Z 2020-07-25T00:07:10Z OWNER

Demo of the new functionality:

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"allow": true for anyone, "allow": false for nobody 665400224  
663779179 https://github.com/simonw/datasette/issues/908#issuecomment-663779179 https://api.github.com/repos/simonw/datasette/issues/908 MDEyOklzc3VlQ29tbWVudDY2Mzc3OTE3OQ== simonw 9599 2020-07-25T00:05:48Z 2020-07-25T00:06:15Z OWNER

The documentation section here now has a bunch of different links to live demos illustrating different "allow" block syntax: https://github.com/simonw/datasette/blob/092874202c8748d6e0d4800eaf707c0145d95ffe/docs/authentication.rst#defining-permissions-with-allow-blocks

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Interactive debugging tool for "allow" blocks 665407663  
663767678 https://github.com/simonw/datasette/issues/906#issuecomment-663767678 https://api.github.com/repos/simonw/datasette/issues/906 MDEyOklzc3VlQ29tbWVudDY2Mzc2NzY3OA== simonw 9599 2020-07-24T23:07:22Z 2020-07-24T23:07:22Z OWNER

Illustration of current system:

https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22terry%22%0D%0A%7D&allow=null - null allows

https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22terry%22%0D%0A%7D&allow={} - {} denies

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"allow": true for anyone, "allow": false for nobody 665400224  
663765308 https://github.com/simonw/datasette/issues/908#issuecomment-663765308 https://api.github.com/repos/simonw/datasette/issues/908 MDEyOklzc3VlQ29tbWVudDY2Mzc2NTMwOA== simonw 9599 2020-07-24T22:57:15Z 2020-07-24T22:57:15Z OWNER

Tool lives at https://latest.datasette.io/-/allow-debug

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Interactive debugging tool for "allow" blocks 665407663  
663764203 https://github.com/simonw/datasette/issues/907#issuecomment-663764203 https://api.github.com/repos/simonw/datasette/issues/907 MDEyOklzc3VlQ29tbWVudDY2Mzc2NDIwMw== simonw 9599 2020-07-24T22:53:07Z 2020-07-24T22:53:07Z OWNER

Actually that is already covered here:
https://github.com/simonw/datasette/blob/6be5654ffab282e8cf39cc138ba2d4496ebc7407/docs/authentication.rst#L158

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow documentation doesn't explain what happens with multiple allow keys 665403403  
663727716 https://github.com/simonw/datasette/pull/901#issuecomment-663727716 https://api.github.com/repos/simonw/datasette/issues/901 MDEyOklzc3VlQ29tbWVudDY2MzcyNzcxNg== codecov[bot] 22429695 2020-07-24T20:47:57Z 2020-07-24T20:47:57Z NONE

Codecov Report

Merging #901 into master will decrease coverage by 0.00%.
The diff coverage is 83.33%.

@@            Coverage Diff             @@
##           master     #901      +/-   ##
==========================================
- Coverage   83.41%   83.41%   -0.01%     
==========================================
  Files          27       27              
  Lines        3636     3642       +6     
==========================================
+ Hits         3033     3038       +5     
- Misses        603      604       +1     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/utils/__init__.py</td> <td>93.76% <66.66%> (-0.18%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/filters.py</td> <td>94.35% <100.00%> (+0.09%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/utils/asgi.py</td> <td>91.47% <0.00%> (+0.07%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d9a5ef1...1285f28. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use None as a default arg 662322234  
663726318 https://github.com/simonw/datasette/issues/908#issuecomment-663726318 https://api.github.com/repos/simonw/datasette/issues/908 MDEyOklzc3VlQ29tbWVudDY2MzcyNjMxOA== simonw 9599 2020-07-24T20:43:57Z 2020-07-24T20:45:38Z OWNER

I can implement this as a plugin. Or it could ship as part of Datasette, somewhere under the /-/ namespace like the PermissionsDebugView and MessagesDebugView tools.

I'm going to ship it in Datasette core, to further reinforce the philosophy that debugging tools are important.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Interactive debugging tool for "allow" blocks 665407663  
663726146 https://github.com/simonw/datasette/issues/907#issuecomment-663726146 https://api.github.com/repos/simonw/datasette/issues/907 MDEyOklzc3VlQ29tbWVudDY2MzcyNjE0Ng== simonw 9599 2020-07-24T20:43:27Z 2020-07-24T20:43:27Z OWNER

It might be good to have a little interactive tool which helps debug these things, since there are quite a few edge-cases and the damage caused if people use them incorrectly is substantial.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow documentation doesn't explain what happens with multiple allow keys 665403403  
663724675 https://github.com/simonw/datasette/issues/456#issuecomment-663724675 https://api.github.com/repos/simonw/datasette/issues/456 MDEyOklzc3VlQ29tbWVudDY2MzcyNDY3NQ== simonw 9599 2020-07-24T20:39:17Z 2020-07-24T20:39:17Z OWNER

Yes this is still a bug!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Installing installs the tests package 442327592  
663724425 https://github.com/simonw/datasette/pull/902#issuecomment-663724425 https://api.github.com/repos/simonw/datasette/issues/902 MDEyOklzc3VlQ29tbWVudDY2MzcyNDQyNQ== simonw 9599 2020-07-24T20:38:42Z 2020-07-24T20:38:42Z OWNER

Thanks for spotting this!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't install tests package 662439034  
663720907 https://github.com/simonw/datasette/issues/906#issuecomment-663720907 https://api.github.com/repos/simonw/datasette/issues/906 MDEyOklzc3VlQ29tbWVudDY2MzcyMDkwNw== simonw 9599 2020-07-24T20:29:24Z 2020-07-24T20:29:24Z OWNER

Here are the existing test cases:
https://github.com/simonw/datasette/blob/2115d7e3457b48b3cf9c81551b9fed2d0e9cd111/tests/test_utils.py#L468-L505

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"allow": true for anyone, "allow": false for nobody 665400224  
663143160 https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-663143160 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48 MDEyOklzc3VlQ29tbWVudDY2MzE0MzE2MA== simonw 9599 2020-07-23T17:46:07Z 2020-07-23T17:46:07Z MEMBER

Frustratingly, these links don't work on PyPI: https://pypi.org/project/twitter-to-sqlite/

There's an issue about that here: https://github.com/pypa/readme_renderer/issues/169

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add a table of contents to the README 663976976  
662630868 https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662630868 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48 MDEyOklzc3VlQ29tbWVudDY2MjYzMDg2OA== simonw 9599 2020-07-22T19:03:02Z 2020-07-22T19:03:02Z MEMBER

Done!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add a table of contents to the README 663976976  
662626901 https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662626901 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48 MDEyOklzc3VlQ29tbWVudDY2MjYyNjkwMQ== simonw 9599 2020-07-22T18:54:53Z 2020-07-22T18:54:53Z MEMBER

I'm going to use a GitHub Action to run npx markdown-toc README.md -i

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add a table of contents to the README 663976976  
662241702 https://github.com/simonw/datasette/issues/905#issuecomment-662241702 https://api.github.com/repos/simonw/datasette/issues/905 MDEyOklzc3VlQ29tbWVudDY2MjI0MTcwMg== simonw 9599 2020-07-22T04:59:46Z 2020-07-22T04:59:46Z OWNER

Deployed and working:

% curl -I 'https://fivethirtyeight.datasettes.com/fivethirtyeight.db'
HTTP/1.1 200 OK
Date: Wed, 22 Jul 2020 04:59:23 GMT
Content-Type: application/octet-stream
Content-Length: 281845760
Connection: keep-alive
Set-Cookie: __cfduid=d550b15c99aa59144e49557ced64fc48a1595393963; expires=Fri, 21-Aug-20 04:59:23 GMT; path=/; domain=.datasettes.com; HttpOnly; SameSite=Lax
Via: 1.1 vegur
Cache-Control: max-age=14400
CF-Cache-Status: MISS
Accept-Ranges: bytes
cf-request-id: 04167d0c7100000540f98e8200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 5b6a978d89b30540-LAX
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
/database.db download should include content-length header 663317875  
662114881 https://github.com/simonw/datasette/issues/905#issuecomment-662114881 https://api.github.com/repos/simonw/datasette/issues/905 MDEyOklzc3VlQ29tbWVudDY2MjExNDg4MQ== simonw 9599 2020-07-21T21:25:37Z 2020-07-21T21:25:37Z OWNER

I can use aiofiles.os.stat for this: https://github.com/Tinche/aiofiles/blob/master/aiofiles/os.py

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
/database.db download should include content-length header 663317875  
661587375 https://github.com/simonw/datasette/pull/902#issuecomment-661587375 https://api.github.com/repos/simonw/datasette/issues/902 MDEyOklzc3VlQ29tbWVudDY2MTU4NzM3NQ== codecov[bot] 22429695 2020-07-21T02:44:49Z 2020-07-21T02:44:49Z NONE

Codecov Report

Merging #902 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #902   +/-   ##
=======================================
  Coverage   83.41%   83.41%           
=======================================
  Files          27       27           
  Lines        3636     3636           
=======================================
  Hits         3033     3033           
  Misses        603      603           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d9a5ef1...9aa139d. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't install tests package 662439034  
661524006 https://github.com/simonw/datasette/issues/456#issuecomment-661524006 https://api.github.com/repos/simonw/datasette/issues/456 MDEyOklzc3VlQ29tbWVudDY2MTUyNDAwNg== abeyerpath 32467826 2020-07-21T01:15:07Z 2020-07-21T01:15:07Z CONTRIBUTOR

Bumping this, as the previous fix is passing the wrong type, and not actually addressing the issue...

The exclude argument needs an iterable of packages instead of a single string (but since str is iterable, it's currently excluding packages t, e, and s.)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Installing installs the tests package 442327592  
660548780 https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660548780 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 MDEyOklzc3VlQ29tbWVudDY2MDU0ODc4MA== simonw 9599 2020-07-18T22:02:37Z 2020-07-18T23:05:56Z MEMBER

https://github-to-sqlite.dogsheep.net/github/tags?_facet=repo

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
github-to-sqlite tags command for fetching tags 660355904  
660551397 https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660551397 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 MDEyOklzc3VlQ29tbWVudDY2MDU1MTM5Nw== simonw 9599 2020-07-18T22:27:32Z 2020-07-18T23:05:45Z MEMBER
with most_recent_releases as (
  with ranked as (
    select
      repo,
      tag_name,
      published_at,
      row_number() OVER (
        partition BY repo
        ORDER BY
          published_at DESC
      ) rank
    FROM
      releases
  )
  select
    *
  from
    ranked
  where
    rank = 1
)
select
  repos.full_name as repo,
  most_recent_releases.tag_name as release,
  commits.committer_date as release_commit_date,
  (
    select
      count(*)
    from
      commits c2
    where
      c2.repo = repos.id
      and c2.committer_date > commits.committer_date
  ) as commits_since_release,
  'https://github.com/' || repos.full_name || '/compare/' || most_recent_releases.tag_name || '...' || repos.default_branch as view_commits
from
  most_recent_releases
  join repos on most_recent_releases.repo = repos.id
  join tags on tags.repo = repos.id
  and tags.name = most_recent_releases.tag_name
  join commits on tags.sha = commits.sha
order by
  commits_since_release desc
<table> <thead> <tr> <th>repo</th> <th>release</th> <th>release_commit_date</th> <th>commits_since_release</th> <th>view_commits</th> </tr> </thead> <tbody> <tr> <td>simonw/datasette</td> <td>0.45</td> <td>2020-07-01T21:43:07Z</td> <td>9</td> <td>https://github.com/simonw/datasette/compare/0.45...master</td> </tr> <tr> <td>dogsheep/twitter-to-sqlite</td> <td>0.21.1</td> <td>2020-04-30T18:20:43Z</td> <td>2</td> <td>https://github.com/dogsheep/twitter-to-sqlite/compare/0.21.1...master</td> </tr> <tr> <td>dogsheep/github-to-sqlite</td> <td>2.3</td> <td>2020-07-09T23:26:34Z</td> <td>2</td> <td>https://github.com/dogsheep/github-to-sqlite/compare/2.3...master</td> </tr> <tr> <td>dogsheep/dogsheep-photos</td> <td>0.4.1</td> <td>2020-05-25T20:11:20Z</td> <td>2</td> <td>https://github.com/dogsheep/dogsheep-photos/compare/0.4.1...master</td> </tr> <tr> <td>dogsheep/swarm-to-sqlite</td> <td>0.3.1</td> <td>2020-03-28T02:29:41Z</td> <td>1</td> <td>https://github.com/dogsheep/swarm-to-sqlite/compare/0.3.1...master</td> </tr> <tr> <td>dogsheep/hacker-news-to-sqlite</td> <td>0.3.1</td> <td>2020-03-21T22:39:34Z</td> <td>1</td> <td>https://github.com/dogsheep/hacker-news-to-sqlite/compare/0.3.1...master</td> </tr> <tr> <td>simonw/sqlite-utils</td> <td>2.11</td> <td>2020-07-08T17:36:07Z</td> <td>0</td> <td>https://github.com/simonw/sqlite-utils/compare/2.11...master</td> </tr> <tr> <td>dogsheep/healthkit-to-sqlite</td> <td>0.5</td> <td>2020-03-28T01:50:51Z</td> <td>0</td> <td>https://github.com/dogsheep/healthkit-to-sqlite/compare/0.5...master</td> </tr> <tr> <td>dogsheep/inaturalist-to-sqlite</td> <td>0.2</td> <td>2020-03-24T00:35:44Z</td> <td>0</td> <td>https://github.com/dogsheep/inaturalist-to-sqlite/compare/0.2...master</td> </tr> <tr> <td>dogsheep/genome-to-sqlite</td> <td>0.1</td> <td>2019-09-19T15:38:10Z</td> <td>0</td> <td>https://github.com/dogsheep/genome-to-sqlite/compare/0.1...master</td> </tr> <tr> <td>dogsheep/pocket-to-sqlite</td> <td>0.2</td> <td>2020-03-27T22:23:16Z</td> <td>0</td> <td>https://github.com/dogsheep/pocket-to-sqlite/compare/0.2...master</td> </tr> </tbody> </table>

https://github-to-sqlite.dogsheep.net/github?sql=with+most_recent_releases+as+%28%0D%0A++with+ranked+as+%28%0D%0A++++select%0D%0A++++++repo%2C%0D%0A++++++tag_name%2C%0D%0A++++++published_at%2C%0D%0A++++++row_number%28%29+OVER+%28%0D%0A++++++++partition+BY+repo%0D%0A++++++++ORDER+BY%0D%0A++++++++++published_at+DESC%0D%0A++++++%29+rank%0D%0A++++FROM%0D%0A++++++releases%0D%0A++%29%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++ranked%0D%0A++where%0D%0A++++rank+%3D+1%0D%0A%29%0D%0Aselect%0D%0A++repos.full_name+as+repo%2C%0D%0A++most_recent_releases.tag_name+as+release%2C%0D%0A++commits.committer_date+as+release_commit_date%2C%0D%0A++%28%0D%0A++++select%0D%0A++++++count%28*%29%0D%0A++++from%0D%0A++++++commits+c2%0D%0A++++where%0D%0A++++++c2.repo+%3D+repos.id%0D%0A++++++and+c2.committer_date+%3E+commits.committer_date%0D%0A++%29+as+commits_since_release%2C%0D%0A++%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos.full_name+%7C%7C+%27%2Fcompare%2F%27+%7C%7C+most_recent_releases.tag_name+%7C%7C+%27...%27+%7C%7C+repos.default_branch+as+view_commits%0D%0Afrom%0D%0A++most_recent_releases%0D%0A++join+repos+on+most_recent_releases.repo+%3D+repos.id%0D%0A++join+tags+on+tags.repo+%3D+repos.id%0D%0A++and+tags.name+%3D+most_recent_releases.tag_name%0D%0A++join+commits+on+tags.sha+%3D+commits.sha%0D%0Aorder+by%0D%0A++commits_since_release+desc

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
github-to-sqlite tags command for fetching tags 660355904  
660554811 https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554811 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 MDEyOklzc3VlQ29tbWVudDY2MDU1NDgxMQ== simonw 9599 2020-07-18T23:03:13Z 2020-07-18T23:03:13Z MEMBER

https://github-to-sqlite.dogsheep.net/github/tags now shows a repo column instead of a repo_id column.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix the demo - it breaks because of the tags table change 660429601  
660554299 https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554299 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 MDEyOklzc3VlQ29tbWVudDY2MDU1NDI5OQ== simonw 9599 2020-07-18T22:58:24Z 2020-07-18T23:02:52Z MEMBER

Deploying the fixed version like this:

$ gcloud config set run/region us-central1
$ gcloud config set project datasette-222320
$ datasette publish cloudrun /tmp/github.db \
            -m demo-metadata.json \
            --service github-to-sqlite \
            --install=py-gfm \
            --install='datasette-search-all>=0.3' \
            --install='datasette-render-markdown>=1.1.2' \
            --install=datasette-pretty-json \
            --install=datasette-json-html \
            --install=datasette-vega
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix the demo - it breaks because of the tags table change 660429601  
660554162 https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554162 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 MDEyOklzc3VlQ29tbWVudDY2MDU1NDE2Mg== simonw 9599 2020-07-18T22:56:58Z 2020-07-18T22:56:58Z MEMBER

Manually fixing the database:

$ wget 'https://github-to-sqlite.dogsheep.net/github.db'
--2020-07-18 15:52:33--  https://github-to-sqlite.dogsheep.net/github.db
Resolving github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)... 172.217.5.115
Connecting to github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)|172.217.5.115|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14626816 (14M) [application/octet-stream]
Saving to: ‘github.db’

github.db                              100%[============================================================================>]  13.95M  1.22MB/s    in 18s     

2020-07-18 15:52:53 (773 KB/s) - ‘github.db’ saved [14626816/14626816]

$ sqlite3 github.db 
SQLite version 3.28.0 2019-04-15 14:49:49
Enter ".help" for usage hints.
sqlite> drop table tags;
sqlite> ^D
$ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils dogsheep/healthkit-to-sqlite dogsheep/swarm-to-sqlite dogsheep/twitter-to-sqlite dogsheep/inaturalist-to-sqlite dogsheep/google-takeout-to-sqlite dogsheep/github-to-sqlite dogsheep/genome-to-sqlite dogsheep/pocket-to-sqlite dogsheep/hacker-news-to-sqlite dogsheep/dogsheep-photos 
$ sqlite-utils tables github.db --counts
[{"table": "users", "count": 4048},
 {"table": "repos", "count": 210},
 ...
 {"table": "stars", "count": 4140},
 {"table": "tags", "count": 188}]
$ sqlite-utils rows github.db tags    
[{"repo": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"},
 {"repo": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"},
 ...
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix the demo - it breaks because of the tags table change 660429601  
660553711 https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553711 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 MDEyOklzc3VlQ29tbWVudDY2MDU1MzcxMQ== simonw 9599 2020-07-18T22:52:16Z 2020-07-18T22:52:16Z MEMBER

I think the best fix is to download the github.db database, manually fix it and then manually deploy it to Cloud Run from my laptop.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix the demo - it breaks because of the tags table change 660429601  
660553646 https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553646 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 MDEyOklzc3VlQ29tbWVudDY2MDU1MzY0Ng== simonw 9599 2020-07-18T22:51:41Z 2020-07-18T22:51:41Z MEMBER

I could fix this by putting REFRESH_DB in a commit message:

https://github.com/dogsheep/github-to-sqlite/blob/4ae4aa6f172344b19ff3513707195ee6d2654bd4/.github/workflows/deploy-demo.yml#L41-L46

But... doing so would lose the data I've collected in https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc concerning the first time each dependent repo was spotted.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix the demo - it breaks because of the tags table change 660429601  
660547502 https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660547502 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 MDEyOklzc3VlQ29tbWVudDY2MDU0NzUwMg== simonw 9599 2020-07-18T21:50:37Z 2020-07-18T21:50:37Z MEMBER
$ github-to-sqlite tags tags.db simonw/datasette dogsheep/github-to-sqlite
$ sqlite-utils tables tags.db --counts
[{"table": "users", "count": 2},
 {"table": "licenses", "count": 1},
 {"table": "repos", "count": 2},
 {"table": "tags", "count": 76},
 {"table": "licenses_fts", "count": 1},
 {"table": "licenses_fts_data", "count": 3},
 {"table": "licenses_fts_idx", "count": 1},
 {"table": "licenses_fts_docsize", "count": 1},
 {"table": "licenses_fts_config", "count": 1},
 {"table": "repos_fts", "count": 2},
 {"table": "repos_fts_data", "count": 3},
 {"table": "repos_fts_idx", "count": 1},
 {"table": "repos_fts_docsize", "count": 2},
 {"table": "repos_fts_config", "count": 1},
 {"table": "users_fts", "count": 2},
 {"table": "users_fts_data", "count": 3},
 {"table": "users_fts_idx", "count": 1},
 {"table": "users_fts_docsize", "count": 2},
 {"table": "users_fts_config", "count": 1}]
$ sqlite-utils rows tags.db tags
[{"repo_id": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"},
 {"repo_id": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"},
 {"repo_id": 107914493, "name": "0.45a4", "sha": "265483173bc8341dc02c8b782b9b59d2ce8bbedb"},
 {"repo_id": 107914493, "name": "0.45a3", "sha": "1f55a4a2b68fa65e56a28baeb7f44122fdeca7e7"},
 {"repo_id": 107914493, "name": "0.45a2", "sha": "1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f"},
 {"repo_id": 107914493, "name": "0.45a1", "sha": "b59b92b1b0517cf18fa748ff9d0a0bf86298dd43"},
 {"repo_id": 107914493, "name": "0.45a0", "sha": "dda932d818b34ccab11730a76554f0a3748d8348"},
 {"repo_id": 107914493, "name": "0.44", "sha": "b906030235efbdff536405d66078f4868ce0d3bd"},
 {"repo_id": 107914493, "name": "0.43", "sha": "d56f402822df102f9cf1a9a056449d01a15e3aae"},
 {"repo_id": 107914493, "name": "0.42", "sha": "af6c6c5d6f929f951c0e63bfd1c82e37a071b50f"},
 {"repo_id": 107914493, "name": "0.41", "sha": "182e5c8745c94576718315f7596ccc81e5e2417b"},
 {"repo_id": 107914493, "name": "0.40", "sha": "8da108193b08abf140716f8ac499f32309dfe9cf"},
 {"repo_id": 107914493, "name": "0.39", "sha": "dedd775512daee49925882654f252df61a9e3b6d"},
 {"repo_id": 107914493, "name": "0.38", "sha": "7e357abbc38dcc9d19a2f1df3252668a48e941e4"},
 {"repo_id": 107914493, "name": "0.37.1", "sha": "be20e6991eac2baa9b43e9b26ae209bae805ede5"},
 {"repo_id": 107914493, "name": "0.37", "sha": "c9e6841482b299fceadc5ad548c2dbf58a8f1227"},
 {"repo_id": 107914493, "name": "0.36", "sha": "b031fe97636b80b05fec409ee1dffb7d044fd4e9"},
 {"repo_id": 107914493, "name": "0.35", "sha": "30b6f71b306a43605c99bef79302ed5cb22d1924"},
 {"repo_id": 107914493, "name": "0.34", "sha": "e7f60d2a9b59752e20de8412f7b0a3e9a5359a31"},
 {"repo_id": 107914493, "name": "0.33", "sha": "59e7014c8a0f4102d7dc79f517540c55c49e1554"},
 {"repo_id": 107914493, "name": "0.32", "sha": "a95bedb9c423fa6d772c93ef47bc40f13a5bea50"},
 {"repo_id": 107914493, "name": "0.31.2", "sha": "b51f258d00bb3c3b401f15d46a1fbd50394dbe1c"},
 {"repo_id": 107914493, "name": "0.31.1", "sha": "a22c7761b61baa61b8e3da7d30887468d61d6b83"},
 {"repo_id": 107914493, "name": "0.31", "sha": "7f89928062b1a1fdb2625a946f7cd5161e597401"},
 {"repo_id": 107914493, "name": "0.30.2", "sha": "2bf7ce5f517d772a16d7855a35a8a75d4456aad7"},
 {"repo_id": 107914493, "name": "0.30.1", "sha": "3ca290e0db03bb4747e24203c445873f74512107"},
 {"repo_id": 107914493, "name": "0.30", "sha": "8050f9e1ece9afd0236ad38c6458c12a4ad917e6"},
 {"repo_id": 107914493, "name": "0.29.3", "sha": "0fc8afde0eb5ef677f4ac31601540d6168c8208d"},
 {"repo_id": 107914493, "name": "0.29.2", "sha": "6abe6faff6b035e9334dd05f8c741ae9b7a47440"},
 {"repo_id": 107914493, "name": "0.29.1", "sha": "2a94f3719fb2c4335fcda374fa92f87272b02d34"},
 {"repo_id": 107914493, "name": "0.29", "sha": "fb7ee8e0ad59a15083234a48e935525f6e7257dd"},
 {"repo_id": 107914493, "name": "0.28", "sha": "e518f76c5f5dd0138032bfb26387f5bb91086a3f"},
 {"repo_id": 107914493, "name": "0.27.1", "sha": "3f3f29ac9afe7c41ffc48a3bd2af473a53eecc8a"},
 {"repo_id": 107914493, "name": "0.27", "sha": "436b8bc1d17c2ab415800ab209204f94e7f7929e"},
 {"repo_id": 107914493, "name": "0.26.2", "sha": "a418c8b44f82d456be523c8690cf7236bb648c22"},
 {"repo_id": 107914493, "name": "0.26.1", "sha": "4722acc73ce761556b18f5dcbe36b7fef2ee2c69"},
 {"repo_id": 107914493, "name": "0.26", "sha": "424e146697309a54c05d5d1ba1f840849ddbafdc"},
 {"repo_id": 107914493, "name": "0.25.2", "sha": "b5128fc53fce6a1bf3b16bad9f318451bc1d1263"},
 {"repo_id": 107914493, "name": "0.25.1", "sha": "3dc0b3fa8c9b9bd81540ffe20c8b7e7a72465274"},
 {"repo_id": 107914493, "name": "0.25", "sha": "57a71377c992753327a16b417daf79df7f506dd1"},
 {"repo_id": 107914493, "name": "0.24", "sha": "28872a1fa789f314b0342f4e6182f1c78d6e2bca"},
 {"repo_id": 107914493, "name": "0.23.2", "sha": "6df6f712b36f0fe75694174906e31242427a8d1d"},
 {"repo_id": 107914493, "name": "0.23.1", "sha": "dea86b9fba78e032ad09673e884e764387daf209"},
 {"repo_id": 107914493, "name": "0.23", "sha": "e04f5b0d348ef7275a0a5ab9eb53527105132885"},
 {"repo_id": 107914493, "name": "0.22.1", "sha": "5d6252788230d168ba09f379d1d2af867e3302ab"},
 {"repo_id": 107914493, "name": "0.22", "sha": "558d9d7bfef3dd633eb16389281b67d42c9bdeef"},
 {"repo_id": 107914493, "name": "0.21", "sha": "403211de632cd15f0820cc9399305fc43c187b47"},
 {"repo_id": 107914493, "name": "0.20", "sha": "3a5d7951ce8f35118ffdd7f8d86e09b909e1218c"},
 {"repo_id": 107914493, "name": "0.19", "sha": "ba9bfa583179c25aaef94b1f44da7eba74620b9a"},
 {"repo_id": 107914493, "name": "0.18", "sha": "43ae15c0d14b3e968e8d5bfef72ac0c39783c3a2"},
 {"repo_id": 107914493, "name": "0.17", "sha": "fb988ace7c7e2bee5ac142a0eab22431d0675a77"},
 {"repo_id": 107914493, "name": "0.16", "sha": "b6539ff04502536bd1fa96e3b1430bdafc456826"},
 {"repo_id": 107914493, "name": "0.15", "sha": "7706fe0c67aba5cfe905c7906cae9e0c43cd75b2"},
 {"repo_id": 107914493, "name": "0.14", "sha": "2edc652df6d786e4f2c3f073e3567002d248be09"},
 {"repo_id": 107914493, "name": "0.13", "sha": "c160f15c3937f8fbe581276f811e8c58f9137bb1"},
 {"repo_id": 107914493, "name": "0.12", "sha": "51bdd67691bd69082ae7690af8b905f06050ee80"},
 {"repo_id": 107914493, "name": "0.11", "sha": "b0f3d4e375655f0764f3137dbcede324f9bbc0cb"},
 {"repo_id": 107914493, "name": "0.10", "sha": "5928c11ee798a232aa4096706cd47e639d1c9fc2"},
 {"repo_id": 107914493, "name": "0.9", "sha": "d75f423b6fcfc074b7c6f8f7679da8876f181edd"},
 {"repo_id": 107914493, "name": "0.8", "sha": "fe279ab7b4ae99dab295d5cf4d39ad06d782997e"},
 {"repo_id": 107914493, "name": "0.7", "sha": "6b3b05b6db0d2a7b7cec8b8dbb4ddc5e12a376b2"},
 {"repo_id": 207052882, "name": "2.3", "sha": "7090e43d804724ef3b31ae5ca9efd6ac05f76cbc"},
 {"repo_id": 207052882, "name": "2.2", "sha": "4fe69783b55465e7692a807d3a02a710f69c9c42"},
 {"repo_id": 207052882, "name": "2.1", "sha": "9d7aed336c8e62bf372caa800cb4aae3985cbae9"},
 {"repo_id": 207052882, "name": "2.0", "sha": "44611df1524a03ce305405e5902c9615e3c73a72"},
 {"repo_id": 207052882, "name": "1.1", "sha": "5cd34bd07d704487d48ac741ee5da5317afe88d2"},
 {"repo_id": 207052882, "name": "1.0.1", "sha": "3b7ab5685de89fcb6fc92d320c0e24b17be05570"},
 {"repo_id": 207052882, "name": "1.0", "sha": "1ea30c8fb1d080bd5e38c577e3ad20bb527a2fe6"},
 {"repo_id": 207052882, "name": "0.7", "sha": "e35eec4343aa560c58c1634cc228d0d46c442304"},
 {"repo_id": 207052882, "name": "0.6", "sha": "9eb737090fafd0e5a7e314be48402374d99e9828"},
 {"repo_id": 207052882, "name": "0.5", "sha": "ae9035f8fe5aff1c54bff4c6b4c2e808a44f0f2a"},
 {"repo_id": 207052882, "name": "0.4", "sha": "8c6251c31a05c58c2bfbef114247642d1b3dbb44"},
 {"repo_id": 207052882, "name": "0.3", "sha": "f697f247468516aa4ee13b1862b59e0dba18d00f"},
 {"repo_id": 207052882, "name": "0.2", "sha": "0fe96bc50fb3d7b1c7e4577db0ddf207eaeebbb9"},
 {"repo_id": 207052882, "name": "0.1.1", "sha": "321e0284c64dc48b2143311009886293c05edb07"},
 {"repo_id": 207052882, "name": "0.1", "sha": "7387c88a3f84704548e81d43b91615c02b61a957"}]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
github-to-sqlite tags command for fetching tags 660355904  
660536265 https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660536265 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 MDEyOklzc3VlQ29tbWVudDY2MDUzNjI2NQ== simonw 9599 2020-07-18T20:15:12Z 2020-07-18T20:15:12Z MEMBER

I want to create a SQL query which shows me all of my repositories that have commits that are NOT in the most recent release.

The releases table doesn't have enough information for this because it doesn't tell you the commit hash associated with each release, just the tag: https://github-to-sqlite.dogsheep.net/github/releases

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
github-to-sqlite tags command for fetching tags 660355904  
660419792 https://github.com/simonw/datasette/issues/898#issuecomment-660419792 https://api.github.com/repos/simonw/datasette/issues/898 MDEyOklzc3VlQ29tbWVudDY2MDQxOTc5Mg== simonw 9599 2020-07-18T03:57:46Z 2020-07-18T03:57:46Z OWNER

This requires some thought. There are various testing utilities that don't exist yet that plugins might benefit from - off the top of my head:

  • assert_permissions_checked
  • assert_template_rendered

I should resist the temptation to provide a reusable version of make_app_client that provides a fully configured Datasette instance because I need to be able to change the design of the Datasette fixtures.db test database without accidentally breaking any plugins that depend on it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.utils.testing module 659873662  
660419499 https://github.com/simonw/datasette/issues/898#issuecomment-660419499 https://api.github.com/repos/simonw/datasette/issues/898 MDEyOklzc3VlQ29tbWVudDY2MDQxOTQ5OQ== simonw 9599 2020-07-18T03:55:13Z 2020-07-18T03:55:13Z OWNER

Maybe I should make httpx a testing dependency of Datasette itself. It's usage is already encouraged in plugins by https://datasette.readthedocs.io/en/stable/testing_plugins.html

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.utils.testing module 659873662  
660318063 https://github.com/simonw/datasette/issues/897#issuecomment-660318063 https://api.github.com/repos/simonw/datasette/issues/897 MDEyOklzc3VlQ29tbWVudDY2MDMxODA2Mw== simonw 9599 2020-07-17T20:16:02Z 2020-07-17T20:16:02Z OWNER

Documentation here: https://datasette.readthedocs.io/en/latest/internals.html#request-object

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Request method for retrieving the unparsed request body 659580487  
659773897 https://github.com/simonw/datasette/issues/896#issuecomment-659773897 https://api.github.com/repos/simonw/datasette/issues/896 MDEyOklzc3VlQ29tbWVudDY1OTc3Mzg5Nw== simonw 9599 2020-07-17T01:26:08Z 2020-07-17T01:26:08Z OWNER

I manually tested it with those plugins and it seems to interoperate just fine - since both of those use <pre> tags for the cases that I care about so they're already expecting white-space to be pre wrapped in some way.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use white-space: pre-wrap on ALL table cell contents 658476055  
659734703 https://github.com/simonw/datasette/issues/896#issuecomment-659734703 https://api.github.com/repos/simonw/datasette/issues/896 MDEyOklzc3VlQ29tbWVudDY1OTczNDcwMw== simonw 9599 2020-07-16T23:34:57Z 2020-07-16T23:34:57Z OWNER

I'm worried about how this will interact with some of the plugins:

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use white-space: pre-wrap on ALL table cell contents 658476055  
659615034 https://github.com/simonw/datasette/issues/896#issuecomment-659615034 https://api.github.com/repos/simonw/datasette/issues/896 MDEyOklzc3VlQ29tbWVudDY1OTYxNTAzNA== simonw 9599 2020-07-16T19:14:07Z 2020-07-16T19:14:07Z OWNER

Demo: https://srccon-2020.datasette.io/srccon?sql=select+id%2C+day%2C+time%2C+event_name%2C+event_description%2C+facilitators+from+sessions+order+by+event_dtstart+limit+101

I really like this:

https://user-images.githubusercontent.com/9599/87712630-d228da00-c75d-11ea-8f44-69886e599a07.png">

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use white-space: pre-wrap on ALL table cell contents 658476055  
659610687 https://github.com/simonw/datasette/issues/896#issuecomment-659610687 https://api.github.com/repos/simonw/datasette/issues/896 MDEyOklzc3VlQ29tbWVudDY1OTYxMDY4Nw== simonw 9599 2020-07-16T19:05:43Z 2020-07-16T19:05:43Z OWNER

I'm going to give this a go - if it turns out to be a bad idea I can revert it back out again.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use white-space: pre-wrap on ALL table cell contents 658476055  
659085528 https://github.com/simonw/datasette/issues/895#issuecomment-659085528 https://api.github.com/repos/simonw/datasette/issues/895 MDEyOklzc3VlQ29tbWVudDY1OTA4NTUyOA== simonw 9599 2020-07-16T00:32:47Z 2020-07-16T00:32:47Z OWNER

This was added in https://github.com/simonw/datasette/commit/504196341c49840270bd75ea1a1871ef386ba7ea - here's the relevant code (which only applies on the table page, not the query page):

https://github.com/simonw/datasette/blob/d6e03b04302a0852e7133dc030eab50177c37be7/datasette/views/table.py#L196-L204

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
SQL query output should show numeric values in a different colour 657747959  
657268433 https://github.com/simonw/datasette/issues/892#issuecomment-657268433 https://api.github.com/repos/simonw/datasette/issues/892 MDEyOklzc3VlQ29tbWVudDY1NzI2ODQzMw== simonw 9599 2020-07-12T20:02:17Z 2020-07-12T20:02:35Z OWNER

Fixed https://datasette.readthedocs.io/en/latest/

https://user-images.githubusercontent.com/9599/87255480-f32cba80-c43f-11ea-8928-01528859929f.png">

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"latest" in new documentation navbar is invisible 655465863  
657268051 https://github.com/simonw/datasette/issues/892#issuecomment-657268051 https://api.github.com/repos/simonw/datasette/issues/892 MDEyOklzc3VlQ29tbWVudDY1NzI2ODA1MQ== simonw 9599 2020-07-12T19:58:24Z 2020-07-12T19:58:24Z OWNER
.wy-side-nav-search > div.version {
    margin-top: -.4045em;
    margin-bottom: .809em;
    font-weight: normal;
    color: rgba(255,255,255,0.3);
}

Fix can go here: https://github.com/simonw/datasette/blob/ee0ef016523a765b6ef6eaa43cad9ad568f78ae4/docs/_static/css/custom.css#L1-L3

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"latest" in new documentation navbar is invisible 655465863  
656363548 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-656363548 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NjM2MzU0OA== simonw 9599 2020-07-09T21:37:28Z 2020-07-09T21:37:28Z OWNER

I'm going to add a second method .transform_table_sql(...) - which returns the SQL that would have been executed but does NOT execute it.

Advanced callers can use this to include their own additional steps in the same transaction - e.g. recreating views or triggers.

More importantly it gives me a useful hook for writing some unit tests against the generated SQL.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740  
655898722 https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655898722 https://api.github.com/repos/simonw/sqlite-utils/issues/121 MDEyOklzc3VlQ29tbWVudDY1NTg5ODcyMg== tsibley 79913 2020-07-09T04:53:08Z 2020-07-09T04:53:08Z CONTRIBUTOR

Yep, I agree that makes more sense for backwards compat and more casual use cases. I think it should be possible for the Database/Queryable methods to DTRT based on seeing if it's within a context-manager-managed transaction.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improved (and better documented) support for transactions 652961907  
655786374 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655786374 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTc4NjM3NA== simonw 9599 2020-07-08T22:16:54Z 2020-07-08T22:16:54Z OWNER

According to https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes the hardest bits to consider are how to deal with existing foreign key relationships, triggers and views.

I'm OK leaving views as an exercise for the caller - many of these transformations may not need any view changes at all.

Foreign key relationships are important: it should handle these automatically as effectively as possible.

Likewise trigger changes: need to think about what this means.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740  
655785396 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655785396 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTc4NTM5Ng== simonw 9599 2020-07-08T22:14:10Z 2020-07-08T22:14:10Z OWNER

Work in progress: not quite right yet, I need smarter logic for how renamed columns are reflected in the generated INSERT INTO ... SELECT ... query:

    def transform_table(
        self,
        columns=None,
        rename=None,
        change_type=None,
        pk=None,
        foreign_keys=None,
        column_order=None,
        not_null=None,
        defaults=None,
        hash_id=None,
        extracts=None,
    ):
        assert self.exists(), "Cannot transform a table that doesn't exist yet"
        columns = columns or self.columns_dict
        if rename is not None or change_type is not None:
            columns = {rename.get(key, key): change_type.get(key, value) for key, value in columns.items()}
        new_table_name = "{}_new_{}".format(self.name, os.urandom(6).hex())
        previous_columns = set(self.columns_dict.keys())
        with self.db.conn:
            columns = {name: value for (name, value) in columns.items()}
            new_table = self.db.create_table(
                new_table_name,
                columns,
                pk=pk,
                foreign_keys=foreign_keys,
                column_order=column_order,
                not_null=not_null,
                defaults=defaults,
                hash_id=hash_id,
                extracts=extracts,
            )
            # Copy across data - but only for columns that exist in both
            new_columns = set(columns.keys())
            columns_to_copy = new_columns.intersection(previous_columns)
            copy_sql = "INSERT INTO [{new_table}] ({new_cols}) SELECT {old_cols} FROM [{old_table}]".format(
                new_table=new_table_name,
                old_table=self.name,
                old_cols=", ".join("[{}]".format(col) for col in columns_to_copy),
                new_cols=", ".join("[{}]".format(rename.get(col, col)) for col in columns_to_copy),
            )
            self.db.conn.execute(copy_sql)
            # Drop the old table
            self.db.conn.execute("DROP TABLE [{}]".format(self.name))
            # Rename the new one
            self.db.conn.execute(
                "ALTER TABLE [{}] RENAME TO [{}]".format(new_table_name, self.name)
            )
        return self
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740  
655783875 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655783875 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTc4Mzg3NQ== simonw 9599 2020-07-08T22:09:51Z 2020-07-08T22:10:16Z OWNER

I can have a convenient change_type={...} parameter for changing column types too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740  
655782477 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655782477 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTc4MjQ3Nw== simonw 9599 2020-07-08T22:06:23Z 2020-07-08T22:06:23Z OWNER

Thinking about the method signature:

    def transform_table(
        self,
        columns,
        pk=None,
        foreign_keys=None,
        column_order=None,
        not_null=None,
        defaults=None,
        hash_id=None,
        extracts=None,
    ):

This requires the caller to provide the exact set of columns for the new table.

It would be useful if this was optional - if you could omit the columns and have it automatically use the previous columns. This would let you change things like the primary key or the column order using the other arguments.

Even better: allow column renaming using an optional rename={...} argument:

db["dogs"].transform_table(rename={"name": "dog_name"})
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740  
655778058 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655778058 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTc3ODA1OA== simonw 9599 2020-07-08T21:54:30Z 2020-07-08T21:54:30Z OWNER

Don't forget this step:

If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740  
655677909 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677909 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTY3NzkwOQ== simonw 9599 2020-07-08T18:16:39Z 2020-07-08T18:16:39Z OWNER

Since neither the term "transform" or "migrate" are used in the codebase at the moment, I think I'll go with .transform_table() - that leaves the term "migrate" available for any future database migrations system (similar to Django's).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740  
655677396 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677396 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTY3NzM5Ng== simonw 9599 2020-07-08T18:15:39Z 2020-07-08T18:15:39Z OWNER

Alternative possible names:
- .transform_table()
- .migrate()
- .transform()

I'm torn between .migrate_table() and .transform_table().

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740  
655677099 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677099 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTY3NzA5OQ== simonw 9599 2020-07-08T18:15:02Z 2020-07-08T18:15:02Z OWNER

I'm not so keen on that chained API - it's pretty complicated.

Here's an idea for a much simpler interface. Essentially it lets you say "take table X and migrate its contents to a new table with this structure - then atomically rename the tables to switch them":

db["mytable"].migrate_table({"id": int, "name": str"}, pk="id")

The migrate_table() method would take the same exact signature as the table.create() method: https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L615-L625

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740  
655674910 https://github.com/simonw/sqlite-utils/issues/119#issuecomment-655674910 https://api.github.com/repos/simonw/sqlite-utils/issues/119 MDEyOklzc3VlQ29tbWVudDY1NTY3NDkxMA== simonw 9599 2020-07-08T18:10:18Z 2020-07-08T18:10:18Z OWNER

This will work similar to how .add_foreign_keys() works: turn on writable_schema and rewrite the sql for that table in the sqlite_master table.

Here's that code today - it could be adapted to include removal of foreign keys that we no longer want:

https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L391-L401

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to remove a foreign key 652700770  
655673896 https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655673896 https://api.github.com/repos/simonw/sqlite-utils/issues/121 MDEyOklzc3VlQ29tbWVudDY1NTY3Mzg5Ng== simonw 9599 2020-07-08T18:08:11Z 2020-07-08T18:08:11Z OWNER

I'm with you on most of this. Completely agreed that the CLI should do everything in a transaction.

The one thing I'm not keen on is forcing calling code to explicitly start a transaction, for a couple of reasons:

  1. It will break all of the existing code out there
  2. It doesn't match to how I most commonly use this library - as an interactive tool in a Jupyter notebook, where I'm generally working against a brand new scratch database and any errors don't actually matter

So... how about this: IF you wrap your code in a with db: block then the .insert() and suchlike methods expect you to manage transactions yourself. But if you don't use the context manager they behave like they do at the moment (or maybe a bit more sensibly).

That way existing code works as it does today, lazy people like me can call .insert() without thinking about transactions, but people writing actual production code (as opposed to Jupyter hacks) have a sensible way to take control of the transactions themselves.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improved (and better documented) support for transactions 652961907  
655653292 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655653292 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTY1MzI5Mg== simonw 9599 2020-07-08T17:26:02Z 2020-07-08T17:26:02Z OWNER

Awesome, thank you very much.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316  
655652679 https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655652679 https://api.github.com/repos/simonw/sqlite-utils/issues/121 MDEyOklzc3VlQ29tbWVudDY1NTY1MjY3OQ== tsibley 79913 2020-07-08T17:24:46Z 2020-07-08T17:24:46Z CONTRIBUTOR

Better transaction handling would be really great. Some of my thoughts on implementing better transaction discipline are in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728.

My preferences:

  • Each CLI command should operate in a single transaction so that either the whole thing succeeds or the whole thing is rolled back. This avoids partially completed operations when an error occurs part way through processing. Partially completed operations are typically much harder to recovery from gracefully and may cause inconsistent data states.

  • The Python API should be transaction-agnostic and rely on the caller to coordinate transactions. Only the caller knows how individual insert, create, update, etc operations/methods should be bundled conceptually into transactions. When the caller is the CLI, for example, that bundling would be at the CLI command-level. Other callers might want to break up operations into multiple transactions. Transactions are usually most useful when controlled at the application-level (like logging configuration) instead of the library level. The library needs to provide an API that's conducive to transaction use, though.

  • The Python API should provide a context manager to provide consistent transactions handling with more useful defaults than Python's sqlite3 module. The latter issues implicit BEGIN statements by default for most DML (INSERT, UPDATE, DELETE, … but not SELECT, I believe), but not DDL (CREATE TABLE, DROP TABLE, CREATE VIEW, …). Notably, the sqlite3 module doesn't issue the implicit BEGIN until the first DML statement. It does not issue it when entering the with conn block, like other DBAPI2-compatible modules do. The with conn block for sqlite3 only arranges to commit or rollback an existing transaction when exiting. Including DDL and SELECTs in transactions is important for operation consistency, though. There are several existing bugs.python.org tickets about this and future changes are in the works, but sqlite-utils can provide its own API sooner. sqlite-utils's Database class could itself be a context manager (built on the sqlite3 connection context manager) which additionally issues an explicit BEGIN when entering. This would then let Python API callers do something like:

db = sqlite_utils.Database(path)

with db: # ← BEGIN issued here by Database.__enter__
    db.insert(…)
    db.create_view(…)
# ← COMMIT/ROLLBACK issue here by sqlite3.connection.__exit__
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improved (and better documented) support for transactions 652961907  
655643078 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655643078 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTY0MzA3OA== tsibley 79913 2020-07-08T17:05:59Z 2020-07-08T17:05:59Z CONTRIBUTOR

The only thing missing from this PR is updates to the documentation.

Ah, yes, thanks for this reminder! I've repushed with doc bits added.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316  
655290625 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655290625 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTI5MDYyNQ== simonw 9599 2020-07-08T05:15:45Z 2020-07-08T05:15:45Z OWNER

Ideally this would all happen in a single transaction, such that other processes talking to the database would not see any inconsistent state while the table copy was taking place. Need to confirm that this is possible. Also refs transactions thoughts in #121.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740  
655289686 https://github.com/simonw/sqlite-utils/pull/120#issuecomment-655289686 https://api.github.com/repos/simonw/sqlite-utils/issues/120 MDEyOklzc3VlQ29tbWVudDY1NTI4OTY4Ng== simonw 9599 2020-07-08T05:13:11Z 2020-07-08T05:13:11Z OWNER

This is an excellent fix, thanks!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix query command's support for DML 652816158  
655286864 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655286864 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTI4Njg2NA== simonw 9599 2020-07-08T05:05:27Z 2020-07-08T05:05:36Z OWNER

The only thing missing from this PR is updates to the documentation. Those need to go in two places:

Here's an example of a previous commit that includes updates to both CLI and API documentation: https://github.com/simonw/sqlite-utils/commit/f9473ace14878212c1fa968b7bd2f51e4f064dba#diff-e3e2a9bfd88566b05001b02a3f51d286

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316  
655284168 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284168 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTI4NDE2OA== simonw 9599 2020-07-08T04:58:00Z 2020-07-08T04:58:00Z OWNER

Oops didn't mean to click "close" there.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316  
655284054 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284054 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTI4NDA1NA== simonw 9599 2020-07-08T04:57:38Z 2020-07-08T04:57:38Z OWNER

Thoughts on transactions would be much appreciated in #121

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316  
655283393 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTI4MzM5Mw== simonw 9599 2020-07-08T04:55:18Z 2020-07-08T04:55:18Z OWNER

This is a really good idea - and thank you for the detailed discussion in the pull request.

I'm keen to discuss how transactions can work better. I tend to use this pattern in my own code:

with db.conn:
    db["table"].insert(...)

But it's not documented and I've not though very hard about it!

I like having inserts that handle 10,000+ rows commit on every chunk so I can watch their progress from another process, but the library should absolutely support people who want to commit all of the rows in a single transaction - or combine changes with DML.

Lots to discuss here. I'll start a new issue.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316  
655239728 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTIzOTcyOA== tsibley 79913 2020-07-08T02:16:42Z 2020-07-08T02:16:42Z CONTRIBUTOR

I fixed my original oops by moving the DELETE FROM $table out of the chunking loop and repushed. I think this change can be considered in isolation from issues around transactions, which I discuss next.

I wanted to make the DELETE + INSERT happen all in the same transaction so it was robust, but that was more complicated than I expected. The transaction handling in the Database/Table classes isn't systematic, and this poses big hurdles to making Table.insert_all (or other operations) consistent and robust in the face of errors.

For example, I wanted to do this (whitespace ignored in diff, so indentation change not highlighted):

diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index d6b9ecf..4107ceb 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -1028,6 +1028,11 @@ class Table(Queryable):
         batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))
         self.last_rowid = None
         self.last_pk = None
+        with self.db.conn:
+            # Explicit BEGIN is necessary because Python's sqlite3 doesn't
+            # issue implicit BEGINs for DDL, only DML.  We mix DDL and DML
+            # below and might execute DDL first, e.g. for table creation.
+            self.db.conn.execute("BEGIN")
             if truncate and self.exists():
                 self.db.conn.execute("DELETE FROM [{}];".format(self.name))
             for chunk in chunks(itertools.chain([first_record], records), batch_size):
@@ -1038,7 +1043,11 @@ class Table(Queryable):
                         # Use the first batch to derive the table names
                         column_types = suggest_column_types(chunk)
                         column_types.update(columns or {})
-                    self.create(
+                        # Not self.create() because that is wrapped in its own
+                        # transaction and Python's sqlite3 doesn't support
+                        # nested transactions.
+                        self.db.create_table(
+                            self.name,
                             column_types,
                             pk,
                             foreign_keys,
@@ -1139,7 +1148,6 @@ class Table(Queryable):
                     flat_values = list(itertools.chain(*values))
                     queries_and_params = [(sql, flat_values)]

-            with self.db.conn:
                 for query, params in queries_and_params:
                     try:
                         result = self.db.conn.execute(query, params)

but that fails in tests because other methods call insert/upsert/insert_all/upsert_all in the middle of their transactions, so the BEGIN statement throws an error (no nested transactions allowed).

Stepping back, it would be nice to make the transaction handling systematic and predictable. One way to do this is to make the sqlite_utils/db.py code generally not begin or commit any transactions, and require the caller to do that instead. This lets the caller mix and match the Python API calls into transactions as appropriate (which is impossible for the API methods themselves to fully determine). Then, make sqlite_utils/cli.py begin and commit a transaction in each @cli.command function, making each command robust and consistent in the face of errors. The big change here, and why I didn't just submit a patch, is that it dramatically changes the Python API to require callers to begin a transaction rather than just immediately calling methods.

There is also the caveat that for each transaction, an explicit BEGIN is also necessary so that DDL as well as DML (as well as SELECTs) are consistent and rolled back on error. There are several bugs.python.org discussions around this particular problem of DDL and some plans to make it better and consistent with DBAPI2, eventually. In the meantime, the sqlite-utils Database class could be a context manager which supports the incantations necessary to do proper transactions. This would still be a Python API change for callers but wouldn't expose them to the weirdness of the sqlite3's default transaction handling.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316  
655052451 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655052451 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTA1MjQ1MQ== tsibley 79913 2020-07-07T18:45:23Z 2020-07-07T18:45:23Z CONTRIBUTOR

Ah, I see the problem. The truncate is inside a loop I didn't realize was there.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 1022.284ms · About: github-to-sqlite