| https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-669241341 |
https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8 |
669241341 |
MDEyOklzc3VlQ29tbWVudDY2OTI0MTM0MQ== |
9599 |
2020-08-05T14:55:14Z |
2020-08-05T14:55:14Z |
MEMBER |
Looks like there's a column that's not consistently there, so the table got created without it.
Easiest fix is to add `alter=True` to this line:
https://github.com/dogsheep/swarm-to-sqlite/blob/f4a82633da927cde672c9d9af92930bfca2e3ddf/swarm_to_sqlite/utils.py#L94
That will cause `sqlite-utils` to notice if there's a missing column and add it. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
648245071 |
|
| https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667585598 |
https://api.github.com/repos/simonw/sqlite-utils/issues/130 |
667585598 |
MDEyOklzc3VlQ29tbWVudDY2NzU4NTU5OA== |
9599 |
2020-08-01T20:51:28Z |
2020-08-01T20:51:28Z |
OWNER |
CLI documentation: https://github.com/simonw/sqlite-utils/commit/57e4eb8e5564af5d97f892b3be8342451ee177a2?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
671130371 |
|
| https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667585561 |
https://api.github.com/repos/simonw/sqlite-utils/issues/130 |
667585561 |
MDEyOklzc3VlQ29tbWVudDY2NzU4NTU2MQ== |
9599 |
2020-08-01T20:50:59Z |
2020-08-01T20:50:59Z |
OWNER |
Turns out it works for FTS4 as well: https://www.sqlite.org/fts3.html#tokenizer |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
671130371 |
|
| https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667584567 |
https://api.github.com/repos/simonw/sqlite-utils/issues/130 |
667584567 |
MDEyOklzc3VlQ29tbWVudDY2NzU4NDU2Nw== |
9599 |
2020-08-01T20:41:09Z |
2020-08-01T20:41:09Z |
OWNER |
API documentation here: https://github.com/simonw/sqlite-utils/commit/617e6f070c85be66ea04c80b78dafd08c875f8c8?short_path=e156262#diff-e1562629b8def6da772d9b0903faf703 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
671130371 |
|
| https://github.com/simonw/datasette/issues/900#issuecomment-667431123 |
https://api.github.com/repos/simonw/datasette/issues/900 |
667431123 |
MDEyOklzc3VlQ29tbWVudDY2NzQzMTEyMw== |
9599 |
2020-07-31T23:56:33Z |
2020-07-31T23:56:33Z |
OWNER |
I think this is the same issue as #865. I'll look at these together! |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
661605489 |
|
| https://github.com/simonw/datasette/issues/899#issuecomment-667430790 |
https://api.github.com/repos/simonw/datasette/issues/899 |
667430790 |
MDEyOklzc3VlQ29tbWVudDY2NzQzMDc5MA== |
9599 |
2020-07-31T23:54:40Z |
2020-07-31T23:54:40Z |
OWNER |
There's no mechanism that can do this at the moment.
You could absolutely support this with a plugin, probably using the `asgi_wrapper` plugin hook. There's an existing package at https://pypi.org/project/asgi-ratelimit/ which may be usable for this - it may even be possible to configure that using https://github.com/simonw/datasette-configure-asgi rather than using it to write a custom plugin.
Using a separate revers proxy would also be a good way to solve this. It depends which option would work best in your environment.
|
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
660827546 |
|
| https://github.com/simonw/datasette/issues/913#issuecomment-667430352 |
https://api.github.com/repos/simonw/datasette/issues/913 |
667430352 |
MDEyOklzc3VlQ29tbWVudDY2NzQzMDM1Mg== |
9599 |
2020-07-31T23:52:10Z |
2020-07-31T23:52:10Z |
OWNER |
The bigger question here is when this mechanism should be used in place of `metadata.json` or `metadata.yml`. Especially since I'm already considering renaming or reworking that mechanism since plugin configuration has nothing to do with database metadata: #493 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
670209331 |
|
| https://github.com/simonw/datasette/issues/913#issuecomment-667429616 |
https://api.github.com/repos/simonw/datasette/issues/913 |
667429616 |
MDEyOklzc3VlQ29tbWVudDY2NzQyOTYxNg== |
9599 |
2020-07-31T23:48:25Z |
2020-07-31T23:49:59Z |
OWNER |
I could let plugins add additional options to `datasette serve` - but what if two plugins both try to register an option with the same name?
A better solution could be to use the existing `--config` option - and allow plugins to register their own, namespaced config options. So you could do things like:
datasette my.db --config datasette-insert:unsafe:1
Maybe even drop the `datasette-` prefix?
datasette my.db --config insert:unsafe:1
I think I prefer keeping the prefix to be honest - it makes it more obvious that this is a setting which comes from a specific named plugin. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
670209331 |
|
| https://github.com/simonw/datasette/issues/913#issuecomment-667429690 |
https://api.github.com/repos/simonw/datasette/issues/913 |
667429690 |
MDEyOklzc3VlQ29tbWVudDY2NzQyOTY5MA== |
9599 |
2020-07-31T23:48:48Z |
2020-07-31T23:48:48Z |
OWNER |
Here's the code in Datasette that parses `--config` options at the moment:
https://github.com/simonw/datasette/blob/7ca8c0521ac1ea48a3cd8d0fe9275d1316e54b43/datasette/cli.py#L25-L40 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
670209331 |
|
| https://github.com/simonw/datasette/issues/849#issuecomment-667424128 |
https://api.github.com/repos/simonw/datasette/issues/849 |
667424128 |
MDEyOklzc3VlQ29tbWVudDY2NzQyNDEyOA== |
9599 |
2020-07-31T23:21:56Z |
2020-07-31T23:23:24Z |
OWNER |
I'm going to change the default branch on the GitHub repository. If something breaks I can always change it back again.
Done that! Default is now `main`. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
639072811 |
|
| https://github.com/simonw/datasette/issues/849#issuecomment-667424020 |
https://api.github.com/repos/simonw/datasette/issues/849 |
667424020 |
MDEyOklzc3VlQ29tbWVudDY2NzQyNDAyMA== |
9599 |
2020-07-31T23:21:30Z |
2020-07-31T23:21:30Z |
OWNER |
https://github.com/simonw/datasette/tree/main branch now exists and will automatically mirror master (and vice-versa). |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
639072811 |
|
| https://github.com/simonw/datasette/issues/849#issuecomment-667295759 |
https://api.github.com/repos/simonw/datasette/issues/849 |
667295759 |
MDEyOklzc3VlQ29tbWVudDY2NzI5NTc1OQ== |
9599 |
2020-07-31T18:45:35Z |
2020-07-31T18:45:35Z |
OWNER |
Watch out for places in the documentation that might link to `master` - e.g. here:
https://github.com/simonw/datasette/blob/2d7fa8b9058dfbf9c7c371cdeec115d32a177dc9/docs/custom_templates.rst#L247 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
639072811 |
|
| https://github.com/simonw/sqlite-utils/issues/124#issuecomment-664105302 |
https://api.github.com/repos/simonw/sqlite-utils/issues/124 |
664105302 |
MDEyOklzc3VlQ29tbWVudDY2NDEwNTMwMg== |
9599 |
2020-07-27T03:54:24Z |
2020-07-30T22:57:51Z |
OWNER |
Documentation: https://github.com/simonw/sqlite-utils/commit/814d4a7f90991be865d38aac45ff12e36df1c67d?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40
> You can pass named parameters to the query using -p:
>
> $ sqlite-utils query dogs.db "select :num * :num2" -p num 5 -p num2 6
> [{":num * :num2": 30}]
|
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665802405 |
|
| https://github.com/simonw/sqlite-utils/issues/129#issuecomment-666752039 |
https://api.github.com/repos/simonw/sqlite-utils/issues/129 |
666752039 |
MDEyOklzc3VlQ29tbWVudDY2Njc1MjAzOQ== |
9599 |
2020-07-30T22:40:55Z |
2020-07-30T22:40:55Z |
OWNER |
This should be a separate command from `insert-files`. SQLite Archives should use a table with this schema:
```sql
CREATE TABLE sqlar(
name TEXT PRIMARY KEY, -- name of the file
mode INT, -- access permissions
mtime INT, -- last modification time
sz INT, -- original file size
data BLOB -- compressed content
);
```
`insert-files` currently treats the table name as a required argument - but it's not necessary for this table. Also there shouldn't be any support for the `--column` option.
So if I write this command it should be this instead:
sqlite-utils sqlar files.db file.txt file2.txt
But at that point, why bother? Users can use `sqlite3 files.db -Ac *.txt` instead.
So I'm not going to bother implementing this. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
668308777 |
|
| https://github.com/simonw/sqlite-utils/issues/127#issuecomment-666063689 |
https://api.github.com/repos/simonw/sqlite-utils/issues/127 |
666063689 |
MDEyOklzc3VlQ29tbWVudDY2NjA2MzY4OQ== |
9599 |
2020-07-30T03:08:51Z |
2020-07-30T03:08:51Z |
OWNER |
Documentation at the bottom of this section: https://github.com/simonw/sqlite-utils/blob/8fe1e6d1be021aeeb8f08b0f77f03b75a83b6f75/docs/cli.rst#inserting-binary-data-from-files |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
666040390 |
|
| https://github.com/simonw/sqlite-utils/issues/127#issuecomment-666047928 |
https://api.github.com/repos/simonw/sqlite-utils/issues/127 |
666047928 |
MDEyOklzc3VlQ29tbWVudDY2NjA0NzkyOA== |
9599 |
2020-07-30T02:31:05Z |
2020-07-30T02:31:05Z |
OWNER |
Maybe could do this using an improved version of this lambda? Could teach it to look for `-` and read from `sys.stdin` if it sees it.
https://github.com/simonw/sqlite-utils/blob/710454d72aed5094573e642344fd075a0ef5372c/sqlite_utils/cli.py#L839 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
666040390 |
|
| https://github.com/simonw/sqlite-utils/issues/129#issuecomment-666046819 |
https://api.github.com/repos/simonw/sqlite-utils/issues/129 |
666046819 |
MDEyOklzc3VlQ29tbWVudDY2NjA0NjgxOQ== |
9599 |
2020-07-30T02:28:34Z |
2020-07-30T02:28:34Z |
OWNER |
This code looks useful as inspiration: https://github.com/j4mie/sqlsite/blob/f2dadb8db5ed7880f8872b6591d8cb1487f777ea/sqlsite/sqlar.py |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
668308777 |
|
| https://github.com/simonw/datasette/issues/909#issuecomment-666010395 |
https://api.github.com/repos/simonw/datasette/issues/909 |
666010395 |
MDEyOklzc3VlQ29tbWVudDY2NjAxMDM5NQ== |
9599 |
2020-07-30T00:56:17Z |
2020-07-30T00:56:17Z |
OWNER |
```
$ curl -I https://latest.datasette.io/fixtures.db
HTTP/1.1 200 OK
content-disposition: attachment; filename="fixtures.db"
content-type: application/octet-stream
Date: Thu, 30 Jul 2020 00:56:05 GMT
Server: Google Frontend
Transfer-Encoding: chunked
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
667467128 |
|
| https://github.com/simonw/datasette/issues/909#issuecomment-665854704 |
https://api.github.com/repos/simonw/datasette/issues/909 |
665854704 |
MDEyOklzc3VlQ29tbWVudDY2NTg1NDcwNA== |
9599 |
2020-07-29T19:22:31Z |
2020-07-29T19:22:31Z |
OWNER |
I think this results in a bug where the "download database" link doesn't include the correct filename: https://github.com/simonw/datasette/blob/549b1c2063db48c4622ee5c7b478a1e3cbc1ac07/datasette/views/database.py#L110-L131 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
667467128 |
|
| https://github.com/simonw/datasette/pull/910#issuecomment-665663131 |
https://api.github.com/repos/simonw/datasette/issues/910 |
665663131 |
MDEyOklzc3VlQ29tbWVudDY2NTY2MzEzMQ== |
22429695 |
2020-07-29T13:26:14Z |
2020-07-29T13:26:14Z |
NONE |
# [Codecov](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=h1) Report
> Merging [#910](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/3c33b421320c0be81a625ca7307b2e4416a9ed5b&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #910 +/- ##
=======================================
Coverage 83.55% 83.55%
=======================================
Files 27 27
Lines 3666 3666
=======================================
Hits 3063 3063
Misses 603 603
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=footer). Last update [3c33b42...3493915](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
|
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
667840539 |
|
| https://github.com/simonw/sqlite-utils/issues/128#issuecomment-664683608 |
https://api.github.com/repos/simonw/sqlite-utils/issues/128 |
664683608 |
MDEyOklzc3VlQ29tbWVudDY2NDY4MzYwOA== |
9599 |
2020-07-27T23:09:22Z |
2020-07-27T23:09:22Z |
OWNER |
This seems to work, but needs more tests:
```diff
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index d6b9ecf..ee26433 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -7,6 +7,7 @@ import itertools
import json
import os
import pathlib
+import uuid
SQLITE_MAX_VARS = 999
@@ -40,11 +41,13 @@ COLUMN_TYPE_MAPPING = {
str: "TEXT",
bytes.__class__: "BLOB",
bytes: "BLOB",
+ memoryview: "BLOB",
datetime.datetime: "TEXT",
datetime.date: "TEXT",
datetime.time: "TEXT",
decimal.Decimal: "FLOAT",
None.__class__: "TEXT",
+ uuid.UUID: "TEXT",
# SQLite explicit types
"TEXT": "TEXT",
"INTEGER": "INTEGER",
@@ -1336,6 +1339,8 @@ def jsonify_if_needed(value):
return json.dumps(value, default=repr)
elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)):
return value.isoformat()
+ elif isinstance(value, uuid.UUID):
+ return str(value)
else:
return value
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
666639051 |
|
| https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664163524 |
https://api.github.com/repos/simonw/sqlite-utils/issues/122 |
664163524 |
MDEyOklzc3VlQ29tbWVudDY2NDE2MzUyNA== |
9599 |
2020-07-27T07:10:41Z |
2020-07-27T07:10:41Z |
OWNER |
Docs: https://github.com/simonw/sqlite-utils/blob/ebc802f7ff0e640b6ae11ea525290fea0115228c/docs/cli.rst#inserting-binary-data-from-files |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665700495 |
|
| https://github.com/simonw/sqlite-utils/issues/127#issuecomment-664163206 |
https://api.github.com/repos/simonw/sqlite-utils/issues/127 |
664163206 |
MDEyOklzc3VlQ29tbWVudDY2NDE2MzIwNg== |
9599 |
2020-07-27T07:10:05Z |
2020-07-27T07:10:05Z |
OWNER |
I tried to get this working but it was a bit tricky because `-` doesn't behave like a regular `pathlib.Path` - needs a bit more thought on how the implementation would work. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
666040390 |
|
| https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071 |
https://api.github.com/repos/simonw/sqlite-utils/issues/122 |
664128071 |
MDEyOklzc3VlQ29tbWVudDY2NDEyODA3MQ== |
9599 |
2020-07-27T05:30:54Z |
2020-07-27T05:30:54Z |
OWNER |
Inserting files by piping them in should work - but since a filename cannot be derived this will need a `--name blah.gif` option.
cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665700495 |
|
| https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664127741 |
https://api.github.com/repos/simonw/sqlite-utils/issues/122 |
664127741 |
MDEyOklzc3VlQ29tbWVudDY2NDEyNzc0MQ== |
9599 |
2020-07-27T05:29:48Z |
2020-07-27T05:29:48Z |
OWNER |
Test command:
```
sqlite-utils insert-files gifs.db *.gif \
-c filename:filename \
-c filepath:filepath \
-c absolutepath:absolutepath \
-c sha256:sha256 \
-c md5:md5 \
-c content:content \
-c mtime:mtime \
-c ctime:ctime \
-c mtime_iso:mtime_iso \
-c ctime_iso:ctime_iso \
-c size:size \
--pk absolutepath
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665700495 |
|
| https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931279 |
https://api.github.com/repos/simonw/sqlite-utils/issues/122 |
663931279 |
MDEyOklzc3VlQ29tbWVudDY2MzkzMTI3OQ== |
9599 |
2020-07-26T03:33:23Z |
2020-07-27T04:30:49Z |
OWNER |
One idea: `sqlite-utils insert-files`
It could work something like this:
sqlite-utils insert-files files.db /tmp/blah.jpg /tmp/foo.gif \
--table files \
-c key:filename -c hash:sha256 -c body:content \
--pk key
This would insert those two image files into the database in a table called `files` with a schema that looks something like this:
```sql
CREATE TABLE files (
key text primary key,
hash text,
body blob
);
```
The `-c key:filename` options here are the most interesting: they let you create the table with a specific layout. The bit before the `:` is the column name. The bit after the `:` can be a range of different things:
- `filename` - just the filename
- `filepath` - the full filepath (provided on the command-line)
- `absolutepath` - the filepath expanded to start with `/home/...` or whatever
- `sha256` - the SHA256 of the contents
- `md5` - the MD5
- `content` - the binary content itself
- `mtime` - the mtime (floating point timestamp)
- `ctime` - the ctime (floating point timestamp)
- `mtime_iso` - the mtime as an ISO datetime
- `ctime_iso` - the mtime as an ISO datetime
- `size` - the size of the file in bytes |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665700495 |
|
| https://github.com/simonw/sqlite-utils/issues/114#issuecomment-664106621 |
https://api.github.com/repos/simonw/sqlite-utils/issues/114 |
664106621 |
MDEyOklzc3VlQ29tbWVudDY2NDEwNjYyMQ== |
9599 |
2020-07-27T04:01:13Z |
2020-07-27T04:01:13Z |
OWNER |
Work in progress in `transform` branch here: https://github.com/simonw/sqlite-utils/tree/transform |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
621989740 |
|
| https://github.com/simonw/sqlite-utils/issues/126#issuecomment-664106405 |
https://api.github.com/repos/simonw/sqlite-utils/issues/126 |
664106405 |
MDEyOklzc3VlQ29tbWVudDY2NDEwNjQwNQ== |
9599 |
2020-07-27T04:00:08Z |
2020-07-27T04:00:33Z |
OWNER |
```
$ echo '[
{
"name": "transparent.gif",
"content": {
"$base64": true,
"encoded": "R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"
}
}
]' | sqlite-utils insert trans.db files - --pk=name
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665819048 |
|
| https://github.com/simonw/sqlite-utils/issues/126#issuecomment-664065597 |
https://api.github.com/repos/simonw/sqlite-utils/issues/126 |
664065597 |
MDEyOklzc3VlQ29tbWVudDY2NDA2NTU5Nw== |
9599 |
2020-07-27T00:51:11Z |
2020-07-27T00:51:11Z |
OWNER |
I'm going to implement this as the reverse of #125 - binary columns in JSON are now output like this:
```json
{
"name": "lorem.txt",
"mode": 33188,
"mtime": 1595805965,
"sz": 16984,
"data": {
"$base64": true,
"encoded": "eJzt0c1xAyEMBeC7q1ABHleR3HxNAQrIjmb4M0gelx+RTY7p4N2WBYT0vmufUknH8kq5lz5pqRFXsTOl3pYkE/NJnHXoStruJEVjc0mOCyTqq/ZMJnXEZW1Js2ZvRm5U+DPKk9hRWqjyvTFx0YfzhT6MpGmN2lR1fzxjyfVMD9dFrS+bnkleMpMam/ZGXgrX1I/K+5Au3S/9lNQRh0k4Gq/RUz8GiKfsQm+7JLsJ6fTo5JhVG00ZU76kZZkxePx49uIjnpNoJyYlWUsoaSl/CcVATje/Kxu13RANnrHweaH3V5Jh4jvGyKCnxJLiXPKhmW3fiCnG7Jql7RR3UvFo8jJ4z039dtOkTFmWzL1be9lt8A5II471m6vXy+l0BR/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984PuP7xubBoN9"
}
}
]
```
So the `sqlite-utils insert` command should learn to spot `{"$base64": true...}` values and base64 decode them before inserting them. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665819048 |
|
| https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664065341 |
https://api.github.com/repos/simonw/sqlite-utils/issues/125 |
664065341 |
MDEyOklzc3VlQ29tbWVudDY2NDA2NTM0MQ== |
9599 |
2020-07-27T00:49:41Z |
2020-07-27T00:49:41Z |
OWNER |
Documentation: https://github.com/simonw/sqlite-utils/commit/20e543e9a492f2e764caae73c38e87f18eaec444?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665817570 |
|
| https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664062546 |
https://api.github.com/repos/simonw/sqlite-utils/issues/125 |
664062546 |
MDEyOklzc3VlQ29tbWVudDY2NDA2MjU0Ng== |
9599 |
2020-07-27T00:33:03Z |
2020-07-27T00:33:03Z |
OWNER |
I'm going to imitate how Datasette solves this problem:
```json
[
{
"name": "lorem.txt",
"mode": 33188,
"mtime": 1595805965,
"sz": 16984,
"data": {
"$base64": true,
"encoded": "eJzt0c1xAyEMBeC7q1ABHleR3HxNAQrIjmb4M0gelx+RTY7p4N2WBYT0vmufUknH8kq5lz5pqRFXsTOl3pYkE/NJnHXoStruJEVjc0mOCyTqq/ZMJnXEZW1Js2ZvRm5U+DPKk9hRWqjyvTFx0YfzhT6MpGmN2lR1fzxjyfVMD9dFrS+bnkleMpMam/ZGXgrX1I/K+5Au3S/9lNQRh0k4Gq/RUz8GiKfsQm+7JLsJ6fTo5JhVG00ZU76kZZkxePx49uIjnpNoJyYlWUsoaSl/CcVATje/Kxu13RANnrHweaH3V5Jh4jvGyKCnxJLiXPKhmW3fiCnG7Jql7RR3UvFo8jJ4z039dtOkTFmWzL1be9lt8A5II471m6vXy+l0BR/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984PuP7xubBoN9"
}
}
]
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665817570 |
|
| https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664048720 |
https://api.github.com/repos/simonw/sqlite-utils/issues/122 |
664048720 |
MDEyOklzc3VlQ29tbWVudDY2NDA0ODcyMA== |
9599 |
2020-07-26T22:32:50Z |
2020-07-26T22:33:20Z |
OWNER |
This seems to work in creating a SQLite archive containing all `.gif` files in the current directory:
/usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 archive.db -A -c *.gif
Then listing files like this:
```
$ /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 archive.db -A -t
copyable.gif
debug-allow.gif
flash.gif
table-md.gif
```
Here's the schema:
```
$ sqlite3 archive.db .schema
CREATE TABLE sqlar(
name TEXT PRIMARY KEY, -- name of the file
mode INT, -- access permissions
mtime INT, -- last modification time
sz INT, -- original file size
data BLOB -- compressed content
);
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665700495 |
|
| https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664048432 |
https://api.github.com/repos/simonw/sqlite-utils/issues/122 |
664048432 |
MDEyOklzc3VlQ29tbWVudDY2NDA0ODQzMg== |
9599 |
2020-07-26T22:29:31Z |
2020-07-26T22:29:31Z |
OWNER |
I'm trying to play with `sqlite3 -A` on my Mac.
`sqlite3 -A` tells me that it's an unknown option - but I used `brew info sqlite` to find my homebrew installed version and it turns out this works:
```
% /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 -A
Wrong number of arguments. Usage:
.archive ... Manage SQL archives
Each command must have exactly one of the following options:
-c, --create Create a new archive
-u, --update Add or update files with changed mtime
-i, --insert Like -u but always add even if unchanged
-t, --list List contents of archive
-x, --extract Extract files from archive
Optional arguments:
-v, --verbose Print each filename as it is processed
-f FILE, --file FILE Use archive FILE (default is current db)
-a FILE, --append FILE Open FILE using the apndvfs VFS
-C DIR, --directory DIR Read/extract files from directory DIR
-n, --dryrun Show the SQL that would have occurred
Examples:
.ar -cf ARCHIVE foo bar # Create ARCHIVE from files foo and bar
.ar -tf ARCHIVE # List members of ARCHIVE
.ar -xvf ARCHIVE # Verbosely extract files from ARCHIVE
See also:
http://sqlite.org/cli.html#sqlar_archive_support
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665700495 |
|
| https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664013338 |
https://api.github.com/repos/simonw/sqlite-utils/issues/122 |
664013338 |
MDEyOklzc3VlQ29tbWVudDY2NDAxMzMzOA== |
9599 |
2020-07-26T16:57:35Z |
2020-07-26T16:57:35Z |
OWNER |
I should consider easy compatibility with https://www.sqlite.org/sqlar.html
> An SQLite Archive is an ordinary SQLite database file that contains the following table as part of its schema:
> ```
> CREATE TABLE sqlar(
> name TEXT PRIMARY KEY, -- name of the file
> mode INT, -- access permissions
> mtime INT, -- last modification time
> sz INT, -- original file size
> data BLOB -- compressed content
> );
> ```
> Each row of the SQLAR table holds the content of a single file. The filename (the full pathname relative to the root of the archive) is in the "name" field. The "mode" field is an integer which is the unix-style access permissions for the file. "mtime" is the modification time of the file in seconds since 1970. "sz" is the original uncompressed size of the file. The "data" field contains the file content. The content is usually compressed using [Deflate](http://zlib.net/), though not always. If the "sz" field is equal to the size of the "data" field, then the content is stored uncompressed. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665700495 |
|
| https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247 |
https://api.github.com/repos/simonw/sqlite-utils/issues/125 |
664012247 |
MDEyOklzc3VlQ29tbWVudDY2NDAxMjI0Nw== |
9599 |
2020-07-26T16:48:46Z |
2020-07-26T16:48:46Z |
OWNER |
I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says "these columns are base64 encoded, store the decoded data in a BLOB".
That would solve inserting binary data using JSON too. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665817570 |
|
| https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012148 |
https://api.github.com/repos/simonw/sqlite-utils/issues/125 |
664012148 |
MDEyOklzc3VlQ29tbWVudDY2NDAxMjE0OA== |
9599 |
2020-07-26T16:47:51Z |
2020-07-26T16:47:51Z |
OWNER |
Best solution I can think of is to return the data as base64. It's a bit nasty since it means you can't round trip it back again. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665817570 |
|
| https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931426 |
https://api.github.com/repos/simonw/sqlite-utils/issues/122 |
663931426 |
MDEyOklzc3VlQ29tbWVudDY2MzkzMTQyNg== |
9599 |
2020-07-26T03:35:58Z |
2020-07-26T16:44:33Z |
OWNER |
Related: #123 (`--raw` option) |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665700495 |
|
| https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931662 |
https://api.github.com/repos/simonw/sqlite-utils/issues/122 |
663931662 |
MDEyOklzc3VlQ29tbWVudDY2MzkzMTY2Mg== |
9599 |
2020-07-26T03:40:29Z |
2020-07-26T03:40:29Z |
OWNER |
Maybe support `--replace` for replacing images with an existing primary key. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665700495 |
|
| https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931317 |
https://api.github.com/repos/simonw/sqlite-utils/issues/122 |
663931317 |
MDEyOklzc3VlQ29tbWVudDY2MzkzMTMxNw== |
9599 |
2020-07-26T03:33:54Z |
2020-07-26T03:33:54Z |
OWNER |
The command also accepts one or more directories, in which case it will recursively scan them for all files that they contain. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665700495 |
|
| https://github.com/simonw/datasette/pull/868#issuecomment-663924220 |
https://api.github.com/repos/simonw/datasette/issues/868 |
663924220 |
MDEyOklzc3VlQ29tbWVudDY2MzkyNDIyMA== |
702729 |
2020-07-26T01:29:00Z |
2020-07-26T01:29:00Z |
NONE |
Ok, so it's been a while but I think I'm making progress. The good news: I have come up with a configuration change for running the tests on Windows on GitHub Actions. The bad news: there's a bunch of extraneous errors on the Windows case. I *think* this is due to Windows file IO and sqlite in a lot of cases, so I'm working through it. I will eventually clean up this PR. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
646448486 |
|
| https://github.com/simonw/datasette/pull/868#issuecomment-650340914 |
https://api.github.com/repos/simonw/datasette/issues/868 |
650340914 |
MDEyOklzc3VlQ29tbWVudDY1MDM0MDkxNA== |
22429695 |
2020-06-26T18:53:02Z |
2020-07-26T01:21:08Z |
NONE |
# [Codecov](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=h1) Report
> Merging [#868](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/a8a5f813722f72703a7aae41135ccc40635cc02f&el=desc) will **increase** coverage by `0.09%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #868 +/- ##
==========================================
+ Coverage 83.31% 83.40% +0.09%
==========================================
Files 27 27
Lines 3595 3634 +39
==========================================
+ Hits 2995 3031 +36
- Misses 600 603 +3
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.99% <0.00%> (-0.29%)` | :arrow_down: |
| [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.37% <0.00%> (-0.08%)` | :arrow_down: |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.39% <0.00%> (-0.05%)` | :arrow_down: |
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.67% <0.00%> (-0.03%)` | :arrow_down: |
| [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <0.00%> (ø)` | |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.93% <0.00%> (+0.05%)` | :arrow_up: |
| [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `81.17% <0.00%> (+3.39%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=footer). Last update [a8a5f81...da21299](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
|
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
646448486 |
|
| https://github.com/simonw/datasette/issues/906#issuecomment-663779460 |
https://api.github.com/repos/simonw/datasette/issues/906 |
663779460 |
MDEyOklzc3VlQ29tbWVudDY2Mzc3OTQ2MA== |
9599 |
2020-07-25T00:07:10Z |
2020-07-25T00:07:10Z |
OWNER |
Demo of the new functionality:
* https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22root%22%0D%0A%7D&allow=false
* https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22root%22%0D%0A%7D&allow=true |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665400224 |
|
| https://github.com/simonw/datasette/issues/908#issuecomment-663779179 |
https://api.github.com/repos/simonw/datasette/issues/908 |
663779179 |
MDEyOklzc3VlQ29tbWVudDY2Mzc3OTE3OQ== |
9599 |
2020-07-25T00:05:48Z |
2020-07-25T00:06:15Z |
OWNER |
The documentation section here now has a bunch of different links to live demos illustrating different "allow" block syntax: https://github.com/simonw/datasette/blob/092874202c8748d6e0d4800eaf707c0145d95ffe/docs/authentication.rst#defining-permissions-with-allow-blocks |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665407663 |
|
| https://github.com/simonw/datasette/issues/906#issuecomment-663767678 |
https://api.github.com/repos/simonw/datasette/issues/906 |
663767678 |
MDEyOklzc3VlQ29tbWVudDY2Mzc2NzY3OA== |
9599 |
2020-07-24T23:07:22Z |
2020-07-24T23:07:22Z |
OWNER |
Illustration of current system:
https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22terry%22%0D%0A%7D&allow=null - `null` allows
https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22terry%22%0D%0A%7D&allow={} - `{}` denies |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665400224 |
|
| https://github.com/simonw/datasette/issues/908#issuecomment-663765308 |
https://api.github.com/repos/simonw/datasette/issues/908 |
663765308 |
MDEyOklzc3VlQ29tbWVudDY2Mzc2NTMwOA== |
9599 |
2020-07-24T22:57:15Z |
2020-07-24T22:57:15Z |
OWNER |
Tool lives at https://latest.datasette.io/-/allow-debug |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665407663 |
|
| https://github.com/simonw/datasette/issues/907#issuecomment-663764203 |
https://api.github.com/repos/simonw/datasette/issues/907 |
663764203 |
MDEyOklzc3VlQ29tbWVudDY2Mzc2NDIwMw== |
9599 |
2020-07-24T22:53:07Z |
2020-07-24T22:53:07Z |
OWNER |
Actually that is already covered here:
https://github.com/simonw/datasette/blob/6be5654ffab282e8cf39cc138ba2d4496ebc7407/docs/authentication.rst#L158 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665403403 |
|
| https://github.com/simonw/datasette/pull/901#issuecomment-663727716 |
https://api.github.com/repos/simonw/datasette/issues/901 |
663727716 |
MDEyOklzc3VlQ29tbWVudDY2MzcyNzcxNg== |
22429695 |
2020-07-24T20:47:57Z |
2020-07-24T20:47:57Z |
NONE |
# [Codecov](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=h1) Report
> Merging [#901](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/d9a5ef1c32a4390e398653ebfd570f8e1a03d93e&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `83.33%`.
[](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #901 +/- ##
==========================================
- Coverage 83.41% 83.41% -0.01%
==========================================
Files 27 27
Lines 3636 3642 +6
==========================================
+ Hits 3033 3038 +5
- Misses 603 604 +1
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/901/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.76% <66.66%> (-0.18%)` | :arrow_down: |
| [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/901/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `94.35% <100.00%> (+0.09%)` | :arrow_up: |
| [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/901/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `91.47% <0.00%> (+0.07%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=footer). Last update [d9a5ef1...1285f28](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
|
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
662322234 |
|
| https://github.com/simonw/datasette/issues/908#issuecomment-663726318 |
https://api.github.com/repos/simonw/datasette/issues/908 |
663726318 |
MDEyOklzc3VlQ29tbWVudDY2MzcyNjMxOA== |
9599 |
2020-07-24T20:43:57Z |
2020-07-24T20:45:38Z |
OWNER |
I can implement this as a plugin. Or it could ship as part of Datasette, somewhere under the `/-/` namespace like the `PermissionsDebugView` and `MessagesDebugView` tools.
I'm going to ship it in Datasette core, to further reinforce the philosophy that debugging tools are important. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665407663 |
|
| https://github.com/simonw/datasette/issues/907#issuecomment-663726146 |
https://api.github.com/repos/simonw/datasette/issues/907 |
663726146 |
MDEyOklzc3VlQ29tbWVudDY2MzcyNjE0Ng== |
9599 |
2020-07-24T20:43:27Z |
2020-07-24T20:43:27Z |
OWNER |
It might be good to have a little interactive tool which helps debug these things, since there are quite a few edge-cases and the damage caused if people use them incorrectly is substantial. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665403403 |
|
| https://github.com/simonw/datasette/issues/456#issuecomment-663724675 |
https://api.github.com/repos/simonw/datasette/issues/456 |
663724675 |
MDEyOklzc3VlQ29tbWVudDY2MzcyNDY3NQ== |
9599 |
2020-07-24T20:39:17Z |
2020-07-24T20:39:17Z |
OWNER |
Yes this is still a bug! |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
442327592 |
|
| https://github.com/simonw/datasette/pull/902#issuecomment-663724425 |
https://api.github.com/repos/simonw/datasette/issues/902 |
663724425 |
MDEyOklzc3VlQ29tbWVudDY2MzcyNDQyNQ== |
9599 |
2020-07-24T20:38:42Z |
2020-07-24T20:38:42Z |
OWNER |
Thanks for spotting this! |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
662439034 |
|
| https://github.com/simonw/datasette/issues/906#issuecomment-663720907 |
https://api.github.com/repos/simonw/datasette/issues/906 |
663720907 |
MDEyOklzc3VlQ29tbWVudDY2MzcyMDkwNw== |
9599 |
2020-07-24T20:29:24Z |
2020-07-24T20:29:24Z |
OWNER |
Here are the existing test cases:
https://github.com/simonw/datasette/blob/2115d7e3457b48b3cf9c81551b9fed2d0e9cd111/tests/test_utils.py#L468-L505 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
665400224 |
|
| https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-663143160 |
https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48 |
663143160 |
MDEyOklzc3VlQ29tbWVudDY2MzE0MzE2MA== |
9599 |
2020-07-23T17:46:07Z |
2020-07-23T17:46:07Z |
MEMBER |
Frustratingly, these links don't work on PyPI: https://pypi.org/project/twitter-to-sqlite/
There's an issue about that here: https://github.com/pypa/readme_renderer/issues/169 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
663976976 |
|
| https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662630868 |
https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48 |
662630868 |
MDEyOklzc3VlQ29tbWVudDY2MjYzMDg2OA== |
9599 |
2020-07-22T19:03:02Z |
2020-07-22T19:03:02Z |
MEMBER |
Done! |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
663976976 |
|
| https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662626901 |
https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48 |
662626901 |
MDEyOklzc3VlQ29tbWVudDY2MjYyNjkwMQ== |
9599 |
2020-07-22T18:54:53Z |
2020-07-22T18:54:53Z |
MEMBER |
I'm going to use a GitHub Action to run `npx markdown-toc README.md -i` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
663976976 |
|
| https://github.com/simonw/datasette/issues/905#issuecomment-662241702 |
https://api.github.com/repos/simonw/datasette/issues/905 |
662241702 |
MDEyOklzc3VlQ29tbWVudDY2MjI0MTcwMg== |
9599 |
2020-07-22T04:59:46Z |
2020-07-22T04:59:46Z |
OWNER |
Deployed and working:
```
% curl -I 'https://fivethirtyeight.datasettes.com/fivethirtyeight.db'
HTTP/1.1 200 OK
Date: Wed, 22 Jul 2020 04:59:23 GMT
Content-Type: application/octet-stream
Content-Length: 281845760
Connection: keep-alive
Set-Cookie: __cfduid=d550b15c99aa59144e49557ced64fc48a1595393963; expires=Fri, 21-Aug-20 04:59:23 GMT; path=/; domain=.datasettes.com; HttpOnly; SameSite=Lax
Via: 1.1 vegur
Cache-Control: max-age=14400
CF-Cache-Status: MISS
Accept-Ranges: bytes
cf-request-id: 04167d0c7100000540f98e8200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 5b6a978d89b30540-LAX
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
663317875 |
|
| https://github.com/simonw/datasette/issues/905#issuecomment-662114881 |
https://api.github.com/repos/simonw/datasette/issues/905 |
662114881 |
MDEyOklzc3VlQ29tbWVudDY2MjExNDg4MQ== |
9599 |
2020-07-21T21:25:37Z |
2020-07-21T21:25:37Z |
OWNER |
I can use `aiofiles.os.stat` for this: https://github.com/Tinche/aiofiles/blob/master/aiofiles/os.py |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
663317875 |
|
| https://github.com/simonw/datasette/pull/902#issuecomment-661587375 |
https://api.github.com/repos/simonw/datasette/issues/902 |
661587375 |
MDEyOklzc3VlQ29tbWVudDY2MTU4NzM3NQ== |
22429695 |
2020-07-21T02:44:49Z |
2020-07-21T02:44:49Z |
NONE |
# [Codecov](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=h1) Report
> Merging [#902](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/d9a5ef1c32a4390e398653ebfd570f8e1a03d93e&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #902 +/- ##
=======================================
Coverage 83.41% 83.41%
=======================================
Files 27 27
Lines 3636 3636
=======================================
Hits 3033 3033
Misses 603 603
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=footer). Last update [d9a5ef1...9aa139d](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
|
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
662439034 |
|
| https://github.com/simonw/datasette/issues/456#issuecomment-661524006 |
https://api.github.com/repos/simonw/datasette/issues/456 |
661524006 |
MDEyOklzc3VlQ29tbWVudDY2MTUyNDAwNg== |
32467826 |
2020-07-21T01:15:07Z |
2020-07-21T01:15:07Z |
CONTRIBUTOR |
Bumping this, as the previous fix is passing the wrong type, and not actually addressing the issue...
The `exclude` argument needs an iterable of packages instead of a single string (but since `str` is iterable, it's currently excluding packages `t`, `e`, and `s`.) |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
442327592 |
|
| https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660548780 |
https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 |
660548780 |
MDEyOklzc3VlQ29tbWVudDY2MDU0ODc4MA== |
9599 |
2020-07-18T22:02:37Z |
2020-07-18T23:05:56Z |
MEMBER |
https://github-to-sqlite.dogsheep.net/github/tags?_facet=repo |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
660355904 |
|
| https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660551397 |
https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 |
660551397 |
MDEyOklzc3VlQ29tbWVudDY2MDU1MTM5Nw== |
9599 |
2020-07-18T22:27:32Z |
2020-07-18T23:05:45Z |
MEMBER |
```sql
with most_recent_releases as (
with ranked as (
select
repo,
tag_name,
published_at,
row_number() OVER (
partition BY repo
ORDER BY
published_at DESC
) rank
FROM
releases
)
select
*
from
ranked
where
rank = 1
)
select
repos.full_name as repo,
most_recent_releases.tag_name as release,
commits.committer_date as release_commit_date,
(
select
count(*)
from
commits c2
where
c2.repo = repos.id
and c2.committer_date > commits.committer_date
) as commits_since_release,
'https://github.com/' || repos.full_name || '/compare/' || most_recent_releases.tag_name || '...' || repos.default_branch as view_commits
from
most_recent_releases
join repos on most_recent_releases.repo = repos.id
join tags on tags.repo = repos.id
and tags.name = most_recent_releases.tag_name
join commits on tags.sha = commits.sha
order by
commits_since_release desc
```
repo | release | release_commit_date | commits_since_release | view_commits
-- | -- | -- | -- | --
simonw/datasette | 0.45 | 2020-07-01T21:43:07Z | 9 | https://github.com/simonw/datasette/compare/0.45...master
dogsheep/twitter-to-sqlite | 0.21.1 | 2020-04-30T18:20:43Z | 2 | https://github.com/dogsheep/twitter-to-sqlite/compare/0.21.1...master
dogsheep/github-to-sqlite | 2.3 | 2020-07-09T23:26:34Z | 2 | https://github.com/dogsheep/github-to-sqlite/compare/2.3...master
dogsheep/dogsheep-photos | 0.4.1 | 2020-05-25T20:11:20Z | 2 | https://github.com/dogsheep/dogsheep-photos/compare/0.4.1...master
dogsheep/swarm-to-sqlite | 0.3.1 | 2020-03-28T02:29:41Z | 1 | https://github.com/dogsheep/swarm-to-sqlite/compare/0.3.1...master
dogsheep/hacker-news-to-sqlite | 0.3.1 | 2020-03-21T22:39:34Z | 1 | https://github.com/dogsheep/hacker-news-to-sqlite/compare/0.3.1...master
simonw/sqlite-utils | 2.11 | 2020-07-08T17:36:07Z | 0 | https://github.com/simonw/sqlite-utils/compare/2.11...master
dogsheep/healthkit-to-sqlite | 0.5 | 2020-03-28T01:50:51Z | 0 | https://github.com/dogsheep/healthkit-to-sqlite/compare/0.5...master
dogsheep/inaturalist-to-sqlite | 0.2 | 2020-03-24T00:35:44Z | 0 | https://github.com/dogsheep/inaturalist-to-sqlite/compare/0.2...master
dogsheep/genome-to-sqlite | 0.1 | 2019-09-19T15:38:10Z | 0 | https://github.com/dogsheep/genome-to-sqlite/compare/0.1...master
dogsheep/pocket-to-sqlite | 0.2 | 2020-03-27T22:23:16Z | 0 | https://github.com/dogsheep/pocket-to-sqlite/compare/0.2...master
https://github-to-sqlite.dogsheep.net/github?sql=with+most_recent_releases+as+%28%0D%0A++with+ranked+as+%28%0D%0A++++select%0D%0A++++++repo%2C%0D%0A++++++tag_name%2C%0D%0A++++++published_at%2C%0D%0A++++++row_number%28%29+OVER+%28%0D%0A++++++++partition+BY+repo%0D%0A++++++++ORDER+BY%0D%0A++++++++++published_at+DESC%0D%0A++++++%29+rank%0D%0A++++FROM%0D%0A++++++releases%0D%0A++%29%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++ranked%0D%0A++where%0D%0A++++rank+%3D+1%0D%0A%29%0D%0Aselect%0D%0A++repos.full_name+as+repo%2C%0D%0A++most_recent_releases.tag_name+as+release%2C%0D%0A++commits.committer_date+as+release_commit_date%2C%0D%0A++%28%0D%0A++++select%0D%0A++++++count%28*%29%0D%0A++++from%0D%0A++++++commits+c2%0D%0A++++where%0D%0A++++++c2.repo+%3D+repos.id%0D%0A++++++and+c2.committer_date+%3E+commits.committer_date%0D%0A++%29+as+commits_since_release%2C%0D%0A++%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos.full_name+%7C%7C+%27%2Fcompare%2F%27+%7C%7C+most_recent_releases.tag_name+%7C%7C+%27...%27+%7C%7C+repos.default_branch+as+view_commits%0D%0Afrom%0D%0A++most_recent_releases%0D%0A++join+repos+on+most_recent_releases.repo+%3D+repos.id%0D%0A++join+tags+on+tags.repo+%3D+repos.id%0D%0A++and+tags.name+%3D+most_recent_releases.tag_name%0D%0A++join+commits+on+tags.sha+%3D+commits.sha%0D%0Aorder+by%0D%0A++commits_since_release+desc |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
660355904 |
|
| https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554811 |
https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 |
660554811 |
MDEyOklzc3VlQ29tbWVudDY2MDU1NDgxMQ== |
9599 |
2020-07-18T23:03:13Z |
2020-07-18T23:03:13Z |
MEMBER |
https://github-to-sqlite.dogsheep.net/github/tags now shows a `repo` column instead of a `repo_id` column. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
660429601 |
|
| https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554299 |
https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 |
660554299 |
MDEyOklzc3VlQ29tbWVudDY2MDU1NDI5OQ== |
9599 |
2020-07-18T22:58:24Z |
2020-07-18T23:02:52Z |
MEMBER |
Deploying the fixed version like this:
```
$ gcloud config set run/region us-central1
$ gcloud config set project datasette-222320
$ datasette publish cloudrun /tmp/github.db \
-m demo-metadata.json \
--service github-to-sqlite \
--install=py-gfm \
--install='datasette-search-all>=0.3' \
--install='datasette-render-markdown>=1.1.2' \
--install=datasette-pretty-json \
--install=datasette-json-html \
--install=datasette-vega
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
660429601 |
|
| https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554162 |
https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 |
660554162 |
MDEyOklzc3VlQ29tbWVudDY2MDU1NDE2Mg== |
9599 |
2020-07-18T22:56:58Z |
2020-07-18T22:56:58Z |
MEMBER |
Manually fixing the database:
```
$ wget 'https://github-to-sqlite.dogsheep.net/github.db'
--2020-07-18 15:52:33-- https://github-to-sqlite.dogsheep.net/github.db
Resolving github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)... 172.217.5.115
Connecting to github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)|172.217.5.115|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14626816 (14M) [application/octet-stream]
Saving to: ‘github.db’
github.db 100%[============================================================================>] 13.95M 1.22MB/s in 18s
2020-07-18 15:52:53 (773 KB/s) - ‘github.db’ saved [14626816/14626816]
$ sqlite3 github.db
SQLite version 3.28.0 2019-04-15 14:49:49
Enter ".help" for usage hints.
sqlite> drop table tags;
sqlite> ^D
$ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils dogsheep/healthkit-to-sqlite dogsheep/swarm-to-sqlite dogsheep/twitter-to-sqlite dogsheep/inaturalist-to-sqlite dogsheep/google-takeout-to-sqlite dogsheep/github-to-sqlite dogsheep/genome-to-sqlite dogsheep/pocket-to-sqlite dogsheep/hacker-news-to-sqlite dogsheep/dogsheep-photos
$ sqlite-utils tables github.db --counts
[{"table": "users", "count": 4048},
{"table": "repos", "count": 210},
...
{"table": "stars", "count": 4140},
{"table": "tags", "count": 188}]
$ sqlite-utils rows github.db tags
[{"repo": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"},
{"repo": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"},
...
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
660429601 |
|
| https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553711 |
https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 |
660553711 |
MDEyOklzc3VlQ29tbWVudDY2MDU1MzcxMQ== |
9599 |
2020-07-18T22:52:16Z |
2020-07-18T22:52:16Z |
MEMBER |
I think the best fix is to download the `github.db` database, manually fix it and then manually deploy it to Cloud Run from my laptop. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
660429601 |
|
| https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553646 |
https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 |
660553646 |
MDEyOklzc3VlQ29tbWVudDY2MDU1MzY0Ng== |
9599 |
2020-07-18T22:51:41Z |
2020-07-18T22:51:41Z |
MEMBER |
I could fix this by putting `REFRESH_DB` in a commit message:
https://github.com/dogsheep/github-to-sqlite/blob/4ae4aa6f172344b19ff3513707195ee6d2654bd4/.github/workflows/deploy-demo.yml#L41-L46
But... doing so would lose the data I've collected in https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc concerning the first time each dependent repo was spotted. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
660429601 |
|
| https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660547502 |
https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 |
660547502 |
MDEyOklzc3VlQ29tbWVudDY2MDU0NzUwMg== |
9599 |
2020-07-18T21:50:37Z |
2020-07-18T21:50:37Z |
MEMBER |
```
$ github-to-sqlite tags tags.db simonw/datasette dogsheep/github-to-sqlite
$ sqlite-utils tables tags.db --counts
[{"table": "users", "count": 2},
{"table": "licenses", "count": 1},
{"table": "repos", "count": 2},
{"table": "tags", "count": 76},
{"table": "licenses_fts", "count": 1},
{"table": "licenses_fts_data", "count": 3},
{"table": "licenses_fts_idx", "count": 1},
{"table": "licenses_fts_docsize", "count": 1},
{"table": "licenses_fts_config", "count": 1},
{"table": "repos_fts", "count": 2},
{"table": "repos_fts_data", "count": 3},
{"table": "repos_fts_idx", "count": 1},
{"table": "repos_fts_docsize", "count": 2},
{"table": "repos_fts_config", "count": 1},
{"table": "users_fts", "count": 2},
{"table": "users_fts_data", "count": 3},
{"table": "users_fts_idx", "count": 1},
{"table": "users_fts_docsize", "count": 2},
{"table": "users_fts_config", "count": 1}]
$ sqlite-utils rows tags.db tags
[{"repo_id": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"},
{"repo_id": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"},
{"repo_id": 107914493, "name": "0.45a4", "sha": "265483173bc8341dc02c8b782b9b59d2ce8bbedb"},
{"repo_id": 107914493, "name": "0.45a3", "sha": "1f55a4a2b68fa65e56a28baeb7f44122fdeca7e7"},
{"repo_id": 107914493, "name": "0.45a2", "sha": "1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f"},
{"repo_id": 107914493, "name": "0.45a1", "sha": "b59b92b1b0517cf18fa748ff9d0a0bf86298dd43"},
{"repo_id": 107914493, "name": "0.45a0", "sha": "dda932d818b34ccab11730a76554f0a3748d8348"},
{"repo_id": 107914493, "name": "0.44", "sha": "b906030235efbdff536405d66078f4868ce0d3bd"},
{"repo_id": 107914493, "name": "0.43", "sha": "d56f402822df102f9cf1a9a056449d01a15e3aae"},
{"repo_id": 107914493, "name": "0.42", "sha": "af6c6c5d6f929f951c0e63bfd1c82e37a071b50f"},
{"repo_id": 107914493, "name": "0.41", "sha": "182e5c8745c94576718315f7596ccc81e5e2417b"},
{"repo_id": 107914493, "name": "0.40", "sha": "8da108193b08abf140716f8ac499f32309dfe9cf"},
{"repo_id": 107914493, "name": "0.39", "sha": "dedd775512daee49925882654f252df61a9e3b6d"},
{"repo_id": 107914493, "name": "0.38", "sha": "7e357abbc38dcc9d19a2f1df3252668a48e941e4"},
{"repo_id": 107914493, "name": "0.37.1", "sha": "be20e6991eac2baa9b43e9b26ae209bae805ede5"},
{"repo_id": 107914493, "name": "0.37", "sha": "c9e6841482b299fceadc5ad548c2dbf58a8f1227"},
{"repo_id": 107914493, "name": "0.36", "sha": "b031fe97636b80b05fec409ee1dffb7d044fd4e9"},
{"repo_id": 107914493, "name": "0.35", "sha": "30b6f71b306a43605c99bef79302ed5cb22d1924"},
{"repo_id": 107914493, "name": "0.34", "sha": "e7f60d2a9b59752e20de8412f7b0a3e9a5359a31"},
{"repo_id": 107914493, "name": "0.33", "sha": "59e7014c8a0f4102d7dc79f517540c55c49e1554"},
{"repo_id": 107914493, "name": "0.32", "sha": "a95bedb9c423fa6d772c93ef47bc40f13a5bea50"},
{"repo_id": 107914493, "name": "0.31.2", "sha": "b51f258d00bb3c3b401f15d46a1fbd50394dbe1c"},
{"repo_id": 107914493, "name": "0.31.1", "sha": "a22c7761b61baa61b8e3da7d30887468d61d6b83"},
{"repo_id": 107914493, "name": "0.31", "sha": "7f89928062b1a1fdb2625a946f7cd5161e597401"},
{"repo_id": 107914493, "name": "0.30.2", "sha": "2bf7ce5f517d772a16d7855a35a8a75d4456aad7"},
{"repo_id": 107914493, "name": "0.30.1", "sha": "3ca290e0db03bb4747e24203c445873f74512107"},
{"repo_id": 107914493, "name": "0.30", "sha": "8050f9e1ece9afd0236ad38c6458c12a4ad917e6"},
{"repo_id": 107914493, "name": "0.29.3", "sha": "0fc8afde0eb5ef677f4ac31601540d6168c8208d"},
{"repo_id": 107914493, "name": "0.29.2", "sha": "6abe6faff6b035e9334dd05f8c741ae9b7a47440"},
{"repo_id": 107914493, "name": "0.29.1", "sha": "2a94f3719fb2c4335fcda374fa92f87272b02d34"},
{"repo_id": 107914493, "name": "0.29", "sha": "fb7ee8e0ad59a15083234a48e935525f6e7257dd"},
{"repo_id": 107914493, "name": "0.28", "sha": "e518f76c5f5dd0138032bfb26387f5bb91086a3f"},
{"repo_id": 107914493, "name": "0.27.1", "sha": "3f3f29ac9afe7c41ffc48a3bd2af473a53eecc8a"},
{"repo_id": 107914493, "name": "0.27", "sha": "436b8bc1d17c2ab415800ab209204f94e7f7929e"},
{"repo_id": 107914493, "name": "0.26.2", "sha": "a418c8b44f82d456be523c8690cf7236bb648c22"},
{"repo_id": 107914493, "name": "0.26.1", "sha": "4722acc73ce761556b18f5dcbe36b7fef2ee2c69"},
{"repo_id": 107914493, "name": "0.26", "sha": "424e146697309a54c05d5d1ba1f840849ddbafdc"},
{"repo_id": 107914493, "name": "0.25.2", "sha": "b5128fc53fce6a1bf3b16bad9f318451bc1d1263"},
{"repo_id": 107914493, "name": "0.25.1", "sha": "3dc0b3fa8c9b9bd81540ffe20c8b7e7a72465274"},
{"repo_id": 107914493, "name": "0.25", "sha": "57a71377c992753327a16b417daf79df7f506dd1"},
{"repo_id": 107914493, "name": "0.24", "sha": "28872a1fa789f314b0342f4e6182f1c78d6e2bca"},
{"repo_id": 107914493, "name": "0.23.2", "sha": "6df6f712b36f0fe75694174906e31242427a8d1d"},
{"repo_id": 107914493, "name": "0.23.1", "sha": "dea86b9fba78e032ad09673e884e764387daf209"},
{"repo_id": 107914493, "name": "0.23", "sha": "e04f5b0d348ef7275a0a5ab9eb53527105132885"},
{"repo_id": 107914493, "name": "0.22.1", "sha": "5d6252788230d168ba09f379d1d2af867e3302ab"},
{"repo_id": 107914493, "name": "0.22", "sha": "558d9d7bfef3dd633eb16389281b67d42c9bdeef"},
{"repo_id": 107914493, "name": "0.21", "sha": "403211de632cd15f0820cc9399305fc43c187b47"},
{"repo_id": 107914493, "name": "0.20", "sha": "3a5d7951ce8f35118ffdd7f8d86e09b909e1218c"},
{"repo_id": 107914493, "name": "0.19", "sha": "ba9bfa583179c25aaef94b1f44da7eba74620b9a"},
{"repo_id": 107914493, "name": "0.18", "sha": "43ae15c0d14b3e968e8d5bfef72ac0c39783c3a2"},
{"repo_id": 107914493, "name": "0.17", "sha": "fb988ace7c7e2bee5ac142a0eab22431d0675a77"},
{"repo_id": 107914493, "name": "0.16", "sha": "b6539ff04502536bd1fa96e3b1430bdafc456826"},
{"repo_id": 107914493, "name": "0.15", "sha": "7706fe0c67aba5cfe905c7906cae9e0c43cd75b2"},
{"repo_id": 107914493, "name": "0.14", "sha": "2edc652df6d786e4f2c3f073e3567002d248be09"},
{"repo_id": 107914493, "name": "0.13", "sha": "c160f15c3937f8fbe581276f811e8c58f9137bb1"},
{"repo_id": 107914493, "name": "0.12", "sha": "51bdd67691bd69082ae7690af8b905f06050ee80"},
{"repo_id": 107914493, "name": "0.11", "sha": "b0f3d4e375655f0764f3137dbcede324f9bbc0cb"},
{"repo_id": 107914493, "name": "0.10", "sha": "5928c11ee798a232aa4096706cd47e639d1c9fc2"},
{"repo_id": 107914493, "name": "0.9", "sha": "d75f423b6fcfc074b7c6f8f7679da8876f181edd"},
{"repo_id": 107914493, "name": "0.8", "sha": "fe279ab7b4ae99dab295d5cf4d39ad06d782997e"},
{"repo_id": 107914493, "name": "0.7", "sha": "6b3b05b6db0d2a7b7cec8b8dbb4ddc5e12a376b2"},
{"repo_id": 207052882, "name": "2.3", "sha": "7090e43d804724ef3b31ae5ca9efd6ac05f76cbc"},
{"repo_id": 207052882, "name": "2.2", "sha": "4fe69783b55465e7692a807d3a02a710f69c9c42"},
{"repo_id": 207052882, "name": "2.1", "sha": "9d7aed336c8e62bf372caa800cb4aae3985cbae9"},
{"repo_id": 207052882, "name": "2.0", "sha": "44611df1524a03ce305405e5902c9615e3c73a72"},
{"repo_id": 207052882, "name": "1.1", "sha": "5cd34bd07d704487d48ac741ee5da5317afe88d2"},
{"repo_id": 207052882, "name": "1.0.1", "sha": "3b7ab5685de89fcb6fc92d320c0e24b17be05570"},
{"repo_id": 207052882, "name": "1.0", "sha": "1ea30c8fb1d080bd5e38c577e3ad20bb527a2fe6"},
{"repo_id": 207052882, "name": "0.7", "sha": "e35eec4343aa560c58c1634cc228d0d46c442304"},
{"repo_id": 207052882, "name": "0.6", "sha": "9eb737090fafd0e5a7e314be48402374d99e9828"},
{"repo_id": 207052882, "name": "0.5", "sha": "ae9035f8fe5aff1c54bff4c6b4c2e808a44f0f2a"},
{"repo_id": 207052882, "name": "0.4", "sha": "8c6251c31a05c58c2bfbef114247642d1b3dbb44"},
{"repo_id": 207052882, "name": "0.3", "sha": "f697f247468516aa4ee13b1862b59e0dba18d00f"},
{"repo_id": 207052882, "name": "0.2", "sha": "0fe96bc50fb3d7b1c7e4577db0ddf207eaeebbb9"},
{"repo_id": 207052882, "name": "0.1.1", "sha": "321e0284c64dc48b2143311009886293c05edb07"},
{"repo_id": 207052882, "name": "0.1", "sha": "7387c88a3f84704548e81d43b91615c02b61a957"}]
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
660355904 |
|
| https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660536265 |
https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 |
660536265 |
MDEyOklzc3VlQ29tbWVudDY2MDUzNjI2NQ== |
9599 |
2020-07-18T20:15:12Z |
2020-07-18T20:15:12Z |
MEMBER |
I want to create a SQL query which shows me all of my repositories that have commits that are NOT in the most recent release.
The releases table doesn't have enough information for this because it doesn't tell you the commit hash associated with each release, just the tag: https://github-to-sqlite.dogsheep.net/github/releases |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
660355904 |
|
| https://github.com/simonw/datasette/issues/898#issuecomment-660419792 |
https://api.github.com/repos/simonw/datasette/issues/898 |
660419792 |
MDEyOklzc3VlQ29tbWVudDY2MDQxOTc5Mg== |
9599 |
2020-07-18T03:57:46Z |
2020-07-18T03:57:46Z |
OWNER |
This requires some thought. There are various testing utilities that don't exist yet that plugins might benefit from - off the top of my head:
- `assert_permissions_checked`
- `assert_template_rendered`
I should resist the temptation to provide a reusable version of `make_app_client` that provides a fully configured Datasette instance because I need to be able to change the design of the Datasette `fixtures.db` test database without accidentally breaking any plugins that depend on it. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
659873662 |
|
| https://github.com/simonw/datasette/issues/898#issuecomment-660419499 |
https://api.github.com/repos/simonw/datasette/issues/898 |
660419499 |
MDEyOklzc3VlQ29tbWVudDY2MDQxOTQ5OQ== |
9599 |
2020-07-18T03:55:13Z |
2020-07-18T03:55:13Z |
OWNER |
Maybe I should make `httpx` a testing dependency of Datasette itself. It's usage is already encouraged in plugins by https://datasette.readthedocs.io/en/stable/testing_plugins.html |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
659873662 |
|
| https://github.com/simonw/datasette/issues/897#issuecomment-660318063 |
https://api.github.com/repos/simonw/datasette/issues/897 |
660318063 |
MDEyOklzc3VlQ29tbWVudDY2MDMxODA2Mw== |
9599 |
2020-07-17T20:16:02Z |
2020-07-17T20:16:02Z |
OWNER |
Documentation here: https://datasette.readthedocs.io/en/latest/internals.html#request-object |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
659580487 |
|
| https://github.com/simonw/datasette/issues/896#issuecomment-659773897 |
https://api.github.com/repos/simonw/datasette/issues/896 |
659773897 |
MDEyOklzc3VlQ29tbWVudDY1OTc3Mzg5Nw== |
9599 |
2020-07-17T01:26:08Z |
2020-07-17T01:26:08Z |
OWNER |
I manually tested it with those plugins and it seems to interoperate just fine - since both of those use `<pre>` tags for the cases that I care about so they're already expecting white-space to be pre wrapped in some way. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
658476055 |
|
| https://github.com/simonw/datasette/issues/896#issuecomment-659734703 |
https://api.github.com/repos/simonw/datasette/issues/896 |
659734703 |
MDEyOklzc3VlQ29tbWVudDY1OTczNDcwMw== |
9599 |
2020-07-16T23:34:57Z |
2020-07-16T23:34:57Z |
OWNER |
I'm worried about how this will interact with some of the plugins:
* https://github.com/simonw/datasette-json-html
* https://github.com/simonw/datasette-pretty-json |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
658476055 |
|
| https://github.com/simonw/datasette/issues/896#issuecomment-659615034 |
https://api.github.com/repos/simonw/datasette/issues/896 |
659615034 |
MDEyOklzc3VlQ29tbWVudDY1OTYxNTAzNA== |
9599 |
2020-07-16T19:14:07Z |
2020-07-16T19:14:07Z |
OWNER |
Demo: https://srccon-2020.datasette.io/srccon?sql=select+id%2C+day%2C+time%2C+event_name%2C+event_description%2C+facilitators+from+sessions+order+by+event_dtstart+limit+101
I really like this:
<img width="1396" alt="srccon__select_id__day__time__event_name__event_description__facilitators_from_sessions_order_by_event_dtstart_limit_101" src="https://user-images.githubusercontent.com/9599/87712630-d228da00-c75d-11ea-8f44-69886e599a07.png">
|
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
658476055 |
|
| https://github.com/simonw/datasette/issues/896#issuecomment-659610687 |
https://api.github.com/repos/simonw/datasette/issues/896 |
659610687 |
MDEyOklzc3VlQ29tbWVudDY1OTYxMDY4Nw== |
9599 |
2020-07-16T19:05:43Z |
2020-07-16T19:05:43Z |
OWNER |
I'm going to give this a go - if it turns out to be a bad idea I can revert it back out again. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
658476055 |
|
| https://github.com/simonw/datasette/issues/895#issuecomment-659085528 |
https://api.github.com/repos/simonw/datasette/issues/895 |
659085528 |
MDEyOklzc3VlQ29tbWVudDY1OTA4NTUyOA== |
9599 |
2020-07-16T00:32:47Z |
2020-07-16T00:32:47Z |
OWNER |
This was added in https://github.com/simonw/datasette/commit/504196341c49840270bd75ea1a1871ef386ba7ea - here's the relevant code (which only applies on the table page, not the query page):
https://github.com/simonw/datasette/blob/d6e03b04302a0852e7133dc030eab50177c37be7/datasette/views/table.py#L196-L204 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
657747959 |
|
| https://github.com/simonw/datasette/issues/892#issuecomment-657268433 |
https://api.github.com/repos/simonw/datasette/issues/892 |
657268433 |
MDEyOklzc3VlQ29tbWVudDY1NzI2ODQzMw== |
9599 |
2020-07-12T20:02:17Z |
2020-07-12T20:02:35Z |
OWNER |
Fixed https://datasette.readthedocs.io/en/latest/
<img width="805" alt="Datasette_—_Datasette_documentation_and_How_to_set_a_custom_theme__·_Issue__638_·_readthedocs_readthedocs_org" src="https://user-images.githubusercontent.com/9599/87255480-f32cba80-c43f-11ea-8928-01528859929f.png">
|
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
655465863 |
|
| https://github.com/simonw/datasette/issues/892#issuecomment-657268051 |
https://api.github.com/repos/simonw/datasette/issues/892 |
657268051 |
MDEyOklzc3VlQ29tbWVudDY1NzI2ODA1MQ== |
9599 |
2020-07-12T19:58:24Z |
2020-07-12T19:58:24Z |
OWNER |
```css
.wy-side-nav-search > div.version {
margin-top: -.4045em;
margin-bottom: .809em;
font-weight: normal;
color: rgba(255,255,255,0.3);
}
```
Fix can go here: https://github.com/simonw/datasette/blob/ee0ef016523a765b6ef6eaa43cad9ad568f78ae4/docs/_static/css/custom.css#L1-L3 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
655465863 |
|
| https://github.com/simonw/sqlite-utils/issues/114#issuecomment-656363548 |
https://api.github.com/repos/simonw/sqlite-utils/issues/114 |
656363548 |
MDEyOklzc3VlQ29tbWVudDY1NjM2MzU0OA== |
9599 |
2020-07-09T21:37:28Z |
2020-07-09T21:37:28Z |
OWNER |
I'm going to add a second method `.transform_table_sql(...)` - which returns the SQL that would have been executed but does NOT execute it.
Advanced callers can use this to include their own additional steps in the same transaction - e.g. recreating views or triggers.
More importantly it gives me a useful hook for writing some unit tests against the generated SQL. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
621989740 |
|
| https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655898722 |
https://api.github.com/repos/simonw/sqlite-utils/issues/121 |
655898722 |
MDEyOklzc3VlQ29tbWVudDY1NTg5ODcyMg== |
79913 |
2020-07-09T04:53:08Z |
2020-07-09T04:53:08Z |
CONTRIBUTOR |
Yep, I agree that makes more sense for backwards compat and more casual use cases. I think it should be possible for the Database/Queryable methods to DTRT based on seeing if it's within a context-manager-managed transaction. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
652961907 |
|
| https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655786374 |
https://api.github.com/repos/simonw/sqlite-utils/issues/114 |
655786374 |
MDEyOklzc3VlQ29tbWVudDY1NTc4NjM3NA== |
9599 |
2020-07-08T22:16:54Z |
2020-07-08T22:16:54Z |
OWNER |
According to https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes the hardest bits to consider are how to deal with existing foreign key relationships, triggers and views.
I'm OK leaving views as an exercise for the caller - many of these transformations may not need any view changes at all.
Foreign key relationships are important: it should handle these automatically as effectively as possible.
Likewise trigger changes: need to think about what this means. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
621989740 |
|
| https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655785396 |
https://api.github.com/repos/simonw/sqlite-utils/issues/114 |
655785396 |
MDEyOklzc3VlQ29tbWVudDY1NTc4NTM5Ng== |
9599 |
2020-07-08T22:14:10Z |
2020-07-08T22:14:10Z |
OWNER |
Work in progress: not quite right yet, I need smarter logic for how renamed columns are reflected in the generated `INSERT INTO ... SELECT ...` query:
```python
def transform_table(
self,
columns=None,
rename=None,
change_type=None,
pk=None,
foreign_keys=None,
column_order=None,
not_null=None,
defaults=None,
hash_id=None,
extracts=None,
):
assert self.exists(), "Cannot transform a table that doesn't exist yet"
columns = columns or self.columns_dict
if rename is not None or change_type is not None:
columns = {rename.get(key, key): change_type.get(key, value) for key, value in columns.items()}
new_table_name = "{}_new_{}".format(self.name, os.urandom(6).hex())
previous_columns = set(self.columns_dict.keys())
with self.db.conn:
columns = {name: value for (name, value) in columns.items()}
new_table = self.db.create_table(
new_table_name,
columns,
pk=pk,
foreign_keys=foreign_keys,
column_order=column_order,
not_null=not_null,
defaults=defaults,
hash_id=hash_id,
extracts=extracts,
)
# Copy across data - but only for columns that exist in both
new_columns = set(columns.keys())
columns_to_copy = new_columns.intersection(previous_columns)
copy_sql = "INSERT INTO [{new_table}] ({new_cols}) SELECT {old_cols} FROM [{old_table}]".format(
new_table=new_table_name,
old_table=self.name,
old_cols=", ".join("[{}]".format(col) for col in columns_to_copy),
new_cols=", ".join("[{}]".format(rename.get(col, col)) for col in columns_to_copy),
)
self.db.conn.execute(copy_sql)
# Drop the old table
self.db.conn.execute("DROP TABLE [{}]".format(self.name))
# Rename the new one
self.db.conn.execute(
"ALTER TABLE [{}] RENAME TO [{}]".format(new_table_name, self.name)
)
return self
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
621989740 |
|
| https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655783875 |
https://api.github.com/repos/simonw/sqlite-utils/issues/114 |
655783875 |
MDEyOklzc3VlQ29tbWVudDY1NTc4Mzg3NQ== |
9599 |
2020-07-08T22:09:51Z |
2020-07-08T22:10:16Z |
OWNER |
I can have a convenient `change_type={...}` parameter for changing column types too. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
621989740 |
|
| https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655782477 |
https://api.github.com/repos/simonw/sqlite-utils/issues/114 |
655782477 |
MDEyOklzc3VlQ29tbWVudDY1NTc4MjQ3Nw== |
9599 |
2020-07-08T22:06:23Z |
2020-07-08T22:06:23Z |
OWNER |
Thinking about the method signature:
```python
def transform_table(
self,
columns,
pk=None,
foreign_keys=None,
column_order=None,
not_null=None,
defaults=None,
hash_id=None,
extracts=None,
):
```
This requires the caller to provide the exact set of columns for the new table.
It would be useful if this was optional - if you could omit the columns and have it automatically use the previous columns. This would let you change things like the primary key or the column order using the other arguments.
Even better: allow column renaming using an optional `rename={...}` argument:
```python
db["dogs"].transform_table(rename={"name": "dog_name"})
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
621989740 |
|
| https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655778058 |
https://api.github.com/repos/simonw/sqlite-utils/issues/114 |
655778058 |
MDEyOklzc3VlQ29tbWVudDY1NTc3ODA1OA== |
9599 |
2020-07-08T21:54:30Z |
2020-07-08T21:54:30Z |
OWNER |
Don't forget this step:
> If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
621989740 |
|
| https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677909 |
https://api.github.com/repos/simonw/sqlite-utils/issues/114 |
655677909 |
MDEyOklzc3VlQ29tbWVudDY1NTY3NzkwOQ== |
9599 |
2020-07-08T18:16:39Z |
2020-07-08T18:16:39Z |
OWNER |
Since neither the term "transform" or "migrate" are used in the codebase at the moment, I think I'll go with `.transform_table()` - that leaves the term "migrate" available for any future database migrations system (similar to Django's). |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
621989740 |
|
| https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677396 |
https://api.github.com/repos/simonw/sqlite-utils/issues/114 |
655677396 |
MDEyOklzc3VlQ29tbWVudDY1NTY3NzM5Ng== |
9599 |
2020-07-08T18:15:39Z |
2020-07-08T18:15:39Z |
OWNER |
Alternative possible names:
- `.transform_table()`
- `.migrate()`
- `.transform()`
I'm torn between `.migrate_table()` and `.transform_table()`. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
621989740 |
|
| https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677099 |
https://api.github.com/repos/simonw/sqlite-utils/issues/114 |
655677099 |
MDEyOklzc3VlQ29tbWVudDY1NTY3NzA5OQ== |
9599 |
2020-07-08T18:15:02Z |
2020-07-08T18:15:02Z |
OWNER |
I'm not so keen on that chained API - it's pretty complicated.
Here's an idea for a much simpler interface. Essentially it lets you say "take table X and migrate its contents to a new table with this structure - then atomically rename the tables to switch them":
```python
db["mytable"].migrate_table({"id": int, "name": str"}, pk="id")
```
The `migrate_table()` method would take the same exact signature as the `table.create()` method: https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L615-L625 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
621989740 |
|
| https://github.com/simonw/sqlite-utils/issues/119#issuecomment-655674910 |
https://api.github.com/repos/simonw/sqlite-utils/issues/119 |
655674910 |
MDEyOklzc3VlQ29tbWVudDY1NTY3NDkxMA== |
9599 |
2020-07-08T18:10:18Z |
2020-07-08T18:10:18Z |
OWNER |
This will work similar to how `.add_foreign_keys()` works: turn on `writable_schema` and rewrite the `sql` for that table in the `sqlite_master` table.
Here's that code today - it could be adapted to include removal of foreign keys that we no longer want:
https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L391-L401 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
652700770 |
|
| https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655673896 |
https://api.github.com/repos/simonw/sqlite-utils/issues/121 |
655673896 |
MDEyOklzc3VlQ29tbWVudDY1NTY3Mzg5Ng== |
9599 |
2020-07-08T18:08:11Z |
2020-07-08T18:08:11Z |
OWNER |
I'm with you on most of this. Completely agreed that the CLI should do everything in a transaction.
The one thing I'm not keen on is forcing calling code to explicitly start a transaction, for a couple of reasons:
1. It will break all of the existing code out there
2. It doesn't match to how I most commonly use this library - as an interactive tool in a Jupyter notebook, where I'm generally working against a brand new scratch database and any errors don't actually matter
So... how about this: IF you wrap your code in a `with db:` block then the `.insert()` and suchlike methods expect you to manage transactions yourself. But if you don't use the context manager they behave like they do at the moment (or maybe a bit more sensibly).
That way existing code works as it does today, lazy people like me can call `.insert()` without thinking about transactions, but people writing actual production code (as opposed to Jupyter hacks) have a sensible way to take control of the transactions themselves. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
652961907 |
|
| https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655653292 |
https://api.github.com/repos/simonw/sqlite-utils/issues/118 |
655653292 |
MDEyOklzc3VlQ29tbWVudDY1NTY1MzI5Mg== |
9599 |
2020-07-08T17:26:02Z |
2020-07-08T17:26:02Z |
OWNER |
Awesome, thank you very much. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
651844316 |
|
| https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655652679 |
https://api.github.com/repos/simonw/sqlite-utils/issues/121 |
655652679 |
MDEyOklzc3VlQ29tbWVudDY1NTY1MjY3OQ== |
79913 |
2020-07-08T17:24:46Z |
2020-07-08T17:24:46Z |
CONTRIBUTOR |
Better transaction handling would be really great. Some of my thoughts on implementing better transaction discipline are in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728.
My preferences:
- Each CLI command should operate in a single transaction so that either the whole thing succeeds or the whole thing is rolled back. This avoids partially completed operations when an error occurs part way through processing. Partially completed operations are typically much harder to recovery from gracefully and may cause inconsistent data states.
- The Python API should be transaction-agnostic and rely on the caller to coordinate transactions. Only the caller knows how individual insert, create, update, etc operations/methods should be bundled conceptually into transactions. When the caller is the CLI, for example, that bundling would be at the CLI command-level. Other callers might want to break up operations into multiple transactions. Transactions are usually most useful when controlled at the application-level (like logging configuration) instead of the library level. The library needs to provide an API that's conducive to transaction use, though.
- The Python API should provide a context manager to provide consistent transactions handling with more useful defaults than Python's `sqlite3` module. The latter issues implicit `BEGIN` statements by default for most DML (`INSERT`, `UPDATE`, `DELETE`, … but not `SELECT`, I believe), but **not** DDL (`CREATE TABLE`, `DROP TABLE`, `CREATE VIEW`, …). Notably, the `sqlite3` module doesn't issue the implicit `BEGIN` until the first DML statement. It _does not_ issue it when entering the `with conn` block, like other DBAPI2-compatible modules do. The `with conn` block for `sqlite3` only arranges to commit or rollback an existing transaction when exiting. Including DDL and `SELECT`s in transactions is important for operation consistency, though. There are several existing bugs.python.org tickets about this and future changes are in the works, but sqlite-utils can provide its own API sooner. sqlite-utils's `Database` class could itself be a context manager (built on the `sqlite3` connection context manager) which additionally issues an explicit `BEGIN` when entering. This would then let Python API callers do something like:
```python
db = sqlite_utils.Database(path)
with db: # ← BEGIN issued here by Database.__enter__
db.insert(…)
db.create_view(…)
# ← COMMIT/ROLLBACK issue here by sqlite3.connection.__exit__
``` |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
652961907 |
|
| https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655643078 |
https://api.github.com/repos/simonw/sqlite-utils/issues/118 |
655643078 |
MDEyOklzc3VlQ29tbWVudDY1NTY0MzA3OA== |
79913 |
2020-07-08T17:05:59Z |
2020-07-08T17:05:59Z |
CONTRIBUTOR |
> The only thing missing from this PR is updates to the documentation.
Ah, yes, thanks for this reminder! I've repushed with doc bits added. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
651844316 |
|
| https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655290625 |
https://api.github.com/repos/simonw/sqlite-utils/issues/114 |
655290625 |
MDEyOklzc3VlQ29tbWVudDY1NTI5MDYyNQ== |
9599 |
2020-07-08T05:15:45Z |
2020-07-08T05:15:45Z |
OWNER |
Ideally this would all happen in a single transaction, such that other processes talking to the database would not see any inconsistent state while the table copy was taking place. Need to confirm that this is possible. Also refs transactions thoughts in #121. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
621989740 |
|
| https://github.com/simonw/sqlite-utils/pull/120#issuecomment-655289686 |
https://api.github.com/repos/simonw/sqlite-utils/issues/120 |
655289686 |
MDEyOklzc3VlQ29tbWVudDY1NTI4OTY4Ng== |
9599 |
2020-07-08T05:13:11Z |
2020-07-08T05:13:11Z |
OWNER |
This is an excellent fix, thanks! |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
652816158 |
|
| https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655286864 |
https://api.github.com/repos/simonw/sqlite-utils/issues/118 |
655286864 |
MDEyOklzc3VlQ29tbWVudDY1NTI4Njg2NA== |
9599 |
2020-07-08T05:05:27Z |
2020-07-08T05:05:36Z |
OWNER |
The only thing missing from this PR is updates to the documentation. Those need to go in two places:
- In the Python API docs. I suggest adding a note to this section about bulk inserts: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/python-api.rst#bulk-inserts
- In the CLI docs, in this section: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/cli.rst#inserting-json-data
Here's an example of a previous commit that includes updates to both CLI and API documentation: https://github.com/simonw/sqlite-utils/commit/f9473ace14878212c1fa968b7bd2f51e4f064dba#diff-e3e2a9bfd88566b05001b02a3f51d286 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
651844316 |
|
| https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284168 |
https://api.github.com/repos/simonw/sqlite-utils/issues/118 |
655284168 |
MDEyOklzc3VlQ29tbWVudDY1NTI4NDE2OA== |
9599 |
2020-07-08T04:58:00Z |
2020-07-08T04:58:00Z |
OWNER |
Oops didn't mean to click "close" there. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
651844316 |
|
| https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284054 |
https://api.github.com/repos/simonw/sqlite-utils/issues/118 |
655284054 |
MDEyOklzc3VlQ29tbWVudDY1NTI4NDA1NA== |
9599 |
2020-07-08T04:57:38Z |
2020-07-08T04:57:38Z |
OWNER |
Thoughts on transactions would be much appreciated in #121 |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
651844316 |
|
| https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393 |
https://api.github.com/repos/simonw/sqlite-utils/issues/118 |
655283393 |
MDEyOklzc3VlQ29tbWVudDY1NTI4MzM5Mw== |
9599 |
2020-07-08T04:55:18Z |
2020-07-08T04:55:18Z |
OWNER |
This is a really good idea - and thank you for the detailed discussion in the pull request.
I'm keen to discuss how transactions can work better. I tend to use this pattern in my own code:
with db.conn:
db["table"].insert(...)
But it's not documented and I've not though very hard about it!
I like having inserts that handle 10,000+ rows commit on every chunk so I can watch their progress from another process, but the library should absolutely support people who want to commit all of the rows in a single transaction - or combine changes with DML.
Lots to discuss here. I'll start a new issue. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
651844316 |
|
| https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728 |
https://api.github.com/repos/simonw/sqlite-utils/issues/118 |
655239728 |
MDEyOklzc3VlQ29tbWVudDY1NTIzOTcyOA== |
79913 |
2020-07-08T02:16:42Z |
2020-07-08T02:16:42Z |
CONTRIBUTOR |
I fixed my original oops by moving the `DELETE FROM $table` out of the chunking loop and repushed. I think this change can be considered in isolation from issues around transactions, which I discuss next.
I wanted to make the DELETE + INSERT happen all in the same transaction so it was robust, but that was more complicated than I expected. The transaction handling in the Database/Table classes isn't systematic, and this poses big hurdles to making `Table.insert_all` (or other operations) consistent and robust in the face of errors.
For example, I wanted to do this (whitespace ignored in diff, so indentation change not highlighted):
```diff
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index d6b9ecf..4107ceb 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -1028,6 +1028,11 @@ class Table(Queryable):
batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))
self.last_rowid = None
self.last_pk = None
+ with self.db.conn:
+ # Explicit BEGIN is necessary because Python's sqlite3 doesn't
+ # issue implicit BEGINs for DDL, only DML. We mix DDL and DML
+ # below and might execute DDL first, e.g. for table creation.
+ self.db.conn.execute("BEGIN")
if truncate and self.exists():
self.db.conn.execute("DELETE FROM [{}];".format(self.name))
for chunk in chunks(itertools.chain([first_record], records), batch_size):
@@ -1038,7 +1043,11 @@ class Table(Queryable):
# Use the first batch to derive the table names
column_types = suggest_column_types(chunk)
column_types.update(columns or {})
- self.create(
+ # Not self.create() because that is wrapped in its own
+ # transaction and Python's sqlite3 doesn't support
+ # nested transactions.
+ self.db.create_table(
+ self.name,
column_types,
pk,
foreign_keys,
@@ -1139,7 +1148,6 @@ class Table(Queryable):
flat_values = list(itertools.chain(*values))
queries_and_params = [(sql, flat_values)]
- with self.db.conn:
for query, params in queries_and_params:
try:
result = self.db.conn.execute(query, params)
```
but that fails in tests because other methods call `insert/upsert/insert_all/upsert_all` in the middle of their transactions, so the BEGIN statement throws an error (no nested transactions allowed).
Stepping back, it would be nice to make the transaction handling systematic and predictable. One way to do this is to make the `sqlite_utils/db.py` code generally not begin or commit any transactions, and require the caller to do that instead. This lets the caller mix and match the Python API calls into transactions as appropriate (which is impossible for the API methods themselves to fully determine). Then, make `sqlite_utils/cli.py` begin and commit a transaction in each `@cli.command` function, making each command robust and consistent in the face of errors. The big change here, and why I didn't just submit a patch, is that it dramatically changes the Python API to _require_ callers to begin a transaction rather than just immediately calling methods.
There is also the caveat that for each transaction, an explicit `BEGIN` is also necessary so that DDL as well as DML (as well as `SELECT`s) are consistent and rolled back on error. There are several bugs.python.org discussions around this particular problem of DDL and some plans to make it better and consistent with DBAPI2, eventually. In the meantime, the sqlite-utils Database class could be a context manager which supports the incantations necessary to do proper transactions. This would still be a Python API change for callers but wouldn't expose them to the weirdness of the sqlite3's default transaction handling. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
651844316 |
|
| https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655052451 |
https://api.github.com/repos/simonw/sqlite-utils/issues/118 |
655052451 |
MDEyOklzc3VlQ29tbWVudDY1NTA1MjQ1MQ== |
79913 |
2020-07-07T18:45:23Z |
2020-07-07T18:45:23Z |
CONTRIBUTOR |
Ah, I see the problem. The truncate is inside a loop I didn't realize was there. |
{
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
651844316 |
|