I've been looking into how to to get this data out of Github (especially now there are "secondary rate limits" without an advertised allowance separate from the regular rate limits.

I've had decent success with the Airbyte github extractor (aside from one data quality issue https://github.com/airbytehq/airbyte/pull/15420 ). Airbyte splits data extraction between the GraphQL and REST endpoints depending on the resource type, but they're very comprehensive.

https://github.com/airbytehq/airbyte/blob/306a75ef5370728e0912cf52a1a898a530db0c90/airbyte-integrations/connectors/source-github/source_github/streams.py#L22-L122

Before this, I tried a few solutions in my own custom wrapper mentioned in this thread + its children https://github.com/PyGithub/PyGithub/issues/1989 , but they weren't working as expected.

I am getting the same error when using github-to-sqlite (which uses sqlite-utils internally). I am also using an M1 MacBook Pro, with macOS Monterey 12.5, with Python 3.10.6 for arm64 installed using pyenv. I have sqlite-utils 3.28 installed.

Codecov Report

Merging #462 (0a14a03) into main (1856002) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #462 +/-

======================================= Coverage 96.64% 96.64%
======================================= Files 6 6
Lines 2566 2566
======================================= Hits 2480 2480
Misses 86 86
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1856002...0a14a03. Read the comment docs.

I created a new link for this - https://discord.gg/Ass7bCAMDw - which should drop people straight into the #sqlite-utils channel in the Datasette Discord.

Codecov Report

Merging #460 (f21f4cc) into main (1856002) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #460 +/-

======================================= Coverage 96.64% 96.64%
======================================= Files 6 6
Lines 2566 2566
======================================= Hits 2480 2480
Misses 86 86
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1856002...f21f4cc. Read the comment docs.

That preview link it added didn't work, maybe because I have a custom domain setup?

Entirely my fault, fixed here :https://github.com/simonw/sqlite-utils/commit/98a28cbfe6cea67f6334b42b74f35b0ddd309561

The neat thing about these is that they can be embedded directly in a GitHub README. Not sure if they show up on PyPI project pages, though I expect that they would.

That preview link it added didn't work, maybe because I have a custom domain setup?

It linked to: https://readthedocs-preview--460.org.readthedocs.build/en/460/ - which 404s

Here's the correct link: https://sqlite-utils--460.org.readthedocs.build/en/460/

I've got a URL shortening plugin that I would like to embed on the query page but I'd like avoid capturing the entire query.html template. A feature like this would solve it. Where's this at and how can I help?

test_dockerfile.py

I'm skipping this for the moment because the new Dockerfile shape introduced in #1249 (comment) isn't compatible with this technique, since it installs Datasette from PyPI rather than directly from the repo.

Will need to change that if I want to do this unit tests thing.

Hello, can't you copy in a later step directly in the output directory, e.g. COPY test_dockerfile.py /usr/lib/python*/.. or something like that ?

I was able to fight through this by capturing the SQL commands from the add_foreign_keys() function in sqlite-utils and then executing them manually via the sqlite3 client, first setting PRAGMA writable_schema on and then updating the sqlite_master table. Still no clue why they were failing when run in context...

Updated documentation: https://github.com/simonw/sqlite-utils/blob/1491b66dd7439dd87cd5cd4c4684f46eb3c5751b/docs/python-api.rst#registering-custom-sql-functions

Here's the build from the PR: https://sqlite-utils--457.org.readthedocs.build/en/457/

Codecov Report

Merging #457 (80b3907) into main (9e6ccea) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #457 +/-

======================================= Coverage 96.64% 96.64%
======================================= Files 6 6
Lines 2566 2566
======================================= Hits 2480 2480
Misses 86 86
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 9e6ccea...80b3907. Read the comment docs.

So code would look something like this:

python try: prefix = rows[-2][sort or sort_desc] except KeyError: # Didn't select sort/sort_desc column - look up value by primary key instead primary_key = rows[-2]["pk"] # But more complex than this prefix = (await db.execute("select * from {table} where pk = ?", [primary_key])).first_value()

So the problem here is that in generating the ?_next= next page link we need the value from the specified sort column - but we're not selecting it any more.

Possible fixes:

• Always include the sort column in the list of columns that are selected, then filter that out before they are displayed
• Use a second query to figure out the _sort or _sort_desc value for that last row, since we know its primary key (we always select primary keys)

Not sure which solution is more elegant. I think it might be the second one.

Traceback: Traceback (most recent call last): File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1264, in route_path response = await view(request, send) File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 134, in view return await self.dispatch_request(request) File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 91, in dispatch_request return await handler(request) File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 361, in get response_or_template_contexts = await self.data(request, **data_kwargs) File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 157, in data return await self._data_traced(request, default_labels, _next, _size) File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 633, in _data_traced prefix = rows[-2][sort or sort_desc] IndexError: No item with that key That's this code here: https://github.com/simonw/datasette/blob/7af67b54b7d9bca43e948510fc62f6db2b748fa8/datasette/views/table.py#L631-L645

I'm using this in a separate FastAPI app, worked perfectly when I changed the AsyncBoundLogger to BoundLogger only.

Also for some reason, I'm now getting some logs surfacing from internal packages, like Elasticsearch. But don't have time to deal with that now.

I keep shipping plugins that set a special hook just so the root user can try them out.

I keep running into a need for this. Every time I create a new plugin that defines a new permission I wish there was a clean way to grant that permission to new users without installing some other permissions plugin.

👋 Any update on getting this merged?

Alternatively, is there a work around for this issue to unblock myself?

edit to add: huge fan of both this project and osxphotos, thanks so much for your work here 🙏 If I had any experience with Python I would offer to help but somehow I've managed to not write any Python in 10+ years of programming 😅

Edit again to add:

Alternatively, is there a work around for this issue to unblock myself?

Yes, there is. I was able to apply the patch of this PR and it applies (mostly) cleanly and works.

• verified I have a high enough version of osxphotos
• downloaded the .patch of this (by appending .patch to the URL
• edited the patch to remove the setup.py changes
• cd to the directory containing dogsheep-photos and git apply 31.patch

Can you add a new test for this? Something derived from the example in #423 would work great.

Wow https://tidyr.tidyverse.org/articles/pivot.html is fascinating.

hadley wickham's melt and reshape could be good inspo: http://had.co.nz/reshape/introduction.pdf

Note that Hadley has since implemented pivot_longer and pivot_wider instead of the previous verbs/functions that he used. Those can be found in the tidyr package and are probably the best reference which includes all of the learnings from years of user feedback. https://tidyr.tidyverse.org/articles/pivot.html

hadley wickham's melt and reshape could be good inspo: http://had.co.nz/reshape/introduction.pdf

for example, i have data on votes that look like this:

| ballot_id | option_id | choice | |-|-|-| | 1 | 1 | 0 | | 1 | 2 | 1 | | 1 | 3 | 0 | | 1 | 4 | 1 | | 2 | 1 | 1 | | 2 | 2 | 0 | | 2 | 3 | 1 | | 2 | 4 | 0 |

and i want to reshape from this long form to this wide form:

| ballot_id | option_id_1 | option_id_2 | option_id_3 | option_id_ 4| |-|-|-|-| -| | 1 | 0 | 1 | 0 | 1 | | 2 | 1 | 0 | 1| 0 |

i could do such a think like this.

sql select ballot_id, sum(choice) filter (where option_id = 1) as option_id_1, sum(choice) filter (where option_id = 2) as option_id_2, sum(choice) filter (where option_id = 3) as option_id_3, sum(choice) filter (where option_id = 4) as option_id_4 from vote group by ballot_id

Really interesting idea! Can you flesh out what this might look like, maybe with a before and after example schema?

Codecov Report

Merging #455 (1b35a92) into main (855bce8) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #455 +/-

======================================= Coverage 96.64% 96.64%
======================================= Files 6 6
Lines 2566 2566
======================================= Hits 2480 2480
Misses 86 86
```

| Impacted Files | Coverage Δ | | |---|---|---| | sqlite_utils/db.py | 97.54% <ø> (ø) | |

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 855bce8...1b35a92. Read the comment docs.

just ran into this!

Documented here: https://docs.datasette.io/en/latest/testing_plugins.html#registering-a-plugin-for-the-duration-of-a-test

That fixed it.

Takes 30m to deploy the demo!

pycmarkgfm doesn't implement the Markdown plugin extension I was using.

I'm going to drop the GFM rendering from the demo, and just treat it as regular markdown.

Here's how the demo is deployed: https://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/.github/workflows/deploy-demo.yml#L103-L119

I'm suspicious of py-gfm, which is used like this:

https://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/demo-metadata.json#L49-L51

https://github.com/Zopieux/py-gfm says that library is no longer maintained, and suggests https://github.com/Zopieux/pycmarkgfm as an alternative.

Thanks!

This is a good idea, thanks.

@dependabot rebase

Documentation for the new hook: https://docs.datasette.io/en/latest/plugin_hooks.html#handle-exception-datasette-request-exception

Keeping this issue open until I've proven the new plugin hook works by releasing a plugin that uses it.

I think this is the right place to move the code to catch Forbidden exceptions (and forwards them to that plugin hook): https://github.com/simonw/datasette/blob/8188f55efc0fcca1be692b0d0c875f2d1ee99f17/datasette/app.py#L1269-L1278

I need to refactor this code so that Forbidden exceptions are handled separately.

Reason is that those already have a plugin hook of their own:

https://github.com/simonw/datasette/blob/8188f55efc0fcca1be692b0d0c875f2d1ee99f17/datasette/app.py#L1384-L1395

My first attempt at this refactored that entire handle_exception method to call the new plugin hook, moving its current implementation into a datasette/handle_exception.py default plugin implementation - but it felt weird having that plugin implementation then itself call the pm.hook.forbidden() hook.

Design decision:

python @hookspec def handle_exception(datasette, request, exception): """Handle an uncaught exception. Can return a Response or None.""" It will also support returning an awaitable, using the await_me_maybe pattern.

This is very strange. GitHub Actions CI here runs against Windows and installs OK.

Marking this as "Help wanted" to see if anyone can figure out what's going on here.

Here are all of the changes I made in this issue: https://github.com/simonw/sqlite-utils/compare/e10536c7f59abbb785f092bf83c4ab94c00e31a3...b9a89a0f2c3559989efe65f25a6e1f8fa76fe8b0

I'm going to skip add-geometry-column and create-spatial-index as well.

Commands that could have --ignore added to them:

• [x] enable-fts
• [x] disable-fts (not doing)
• [x] add-column
• [x] add-foreign-keys (decided not to do this)
• [x] duplicate
• [x] add-geometry-column (decided not to do this)
• [x] create-spatial-index (decided not to do this)

add-foreign-keys is enough of a power-feature that I'm happy not to add this there.

I can do this for --create-index instead: Options: --name TEXT Explicit name for the new index --unique Make this a unique index --if-not-exists, --ignore Ignore if index already exists --analyze Run ANALYZE after creating the index --load-extension TEXT SQLite extensions to load -h, --help Show this message and exit. That supports both names for the option.

create-spatial-index is tricky. For consistency with create-index it should really use --if-not-exists - but under the hood it's not using IF NOT EXISTS because those indexes are created using select CreateSpatialIndex(...).

Really this highlights that --if-not-exists is a bad name for that option on create-index.

I'm tempted to add --ignore to create-index and mark --if-not-exists as deprecated - though it will still work because I don't want to release a major version bump.

disable-fts already fails silently if the table doesn't have FTS setup.

For enable-fts I should use --replace instead, since that matches how the Python API works: https://github.com/simonw/sqlite-utils/blob/9dd4cf891d9f4565019e030ddc712507ac87b998/sqlite_utils/db.py#L2088-L2106

Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-chunks

From https://sqlite-utils.datasette.io/en/latest/cli-reference.html

Commands that currently have an --ignore flag, and their descriptions:

• insert: Ignore records if pk already exists
• create-table: If table already exists, do nothing
• add-foreign-key: If foreign key already exists, do nothing
• drop-table: description is missing
• create-view: If view already exists, do nothing
• drop-view: description is missing

And create-index has:

• --if-not-exists: Ignore if index already exists

Documentation can go here: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite-utils-utils

I should warn you that this isn't a supported API - I reserve the right to change how it works between release without a major version bump, because it's not part of the documented API surface.

You'll be fine if you pin to exact versions of the library though!

You may find this recently-documented function useful though: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file

See: - #443

I'm going to close this issue for the moment, but if anyone wants to submit a PR that cleans up this I'll happily review it.

Documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#duplicating-tables - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#duplicate

Documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#duplicating-tables - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.duplicate

Codecov Report

Merging #452 (eef350f) into main (42440d6) will increase coverage by 0.00%. The diff coverage is 100.00%.

```diff @@ Coverage Diff @@

main #452 +/-

======================================= Coverage 96.60% 96.61%
======================================= Files 6 6
Lines 2534 2540 +6
======================================= + Hits 2448 2454 +6
Misses 86 86
```

| Impacted Files | Coverage Δ | | |---|---|---| | sqlite_utils/db.py | 97.53% <100.00%> (+0.01%) | :arrow_up: |

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 42440d6...eef350f. Read the comment docs.

Had to lookup that Base400 thing: https://github.com/simonw/datasette/blob/950cc7677f65aa2543067b3bbfc2b6acb98b62c8/datasette/utils/asgi.py#L16-L29

In DatasetteRouter I'm going to rename handle_500 to handle_exception because it already deals with error codes other than 500 (Forbidden exceptions become 403 and Base400 become 400).

I'll implement this hook and then release it as 0.62a1.

Returning a Response is by far the most intuitive way to design this though. Plugin authors shouldn't care that DatasetteRouter (an undocumented internal API of Datasette) doesn't currently handle responses.

It's weird to return a Response though because the code in question lives in DatasetteRouter which currently works outside of the layer where responses happen - everything in that class at the moment works using send directly.

... maybe it should take send? But then how would plugins know that another plugin hadn't already used send to send a response, and avoid two trying to send at the same time?

Proposed hook design: python @hookspec def handle_exception(datasette, request, exception): """Handle an uncaught exception""" It takes request in case it needs to render a template and pass the request to it.

I think the hook gets called any time any exception makes it to this function: https://github.com/simonw/datasette/blob/950cc7677f65aa2543067b3bbfc2b6acb98b62c8/datasette/app.py#L1374-L1440

Multiple plugins can register for the hook. If they return a Response then that's returned to the user - if they return None then they can quietly do something with the error (log it to Sentry for example) and let some other handler return a response.

I think Datasette should have a default plugin hook implementation which returns the 500 error page.

Codecov Report

Merging #1769 (bb3c610) into main (950cc76) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #1769 +/-

======================================= Coverage 91.67% 91.67%
======================================= Files 36 36
Lines 4658 4658
======================================= Hits 4270 4270
Misses 388 388
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 950cc76...bb3c610. Read the comment docs.

But do I need to pass the use_rowid boolean here as well, as used by def path_from_row_pks(row, pks, use_rowid, quote=True)? Or can I derive that from the fact that pks is an empty tuple?

The row_path part of these really isn't very user friendly, since you need to properly URL-encode the values. The safest way to do so is by calling this obscure, undocumented utility function: https://github.com/simonw/datasette/blob/f0bd2d05f5f7832df4879822afb99d2096c00d48/datasette/utils/init.py#L84-L98

This feels like it should be improved before I turn it into a documented API.

(Note that this API deals with a row that is a potentially-nested dictionary - not with a sqlite3.Row object.)

I think the best way to do this is to change the method signatures to: ```python def row(self, database, table, row, pks): ...

def row_blob(self, database, table, row, pks, column): ... ```

If I do require pks to be passed here, maybe I could make those available to the render_cell() plugin hook to at least make this a bit more pleasant for plugin authors to use?

Current hook: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-row-value-column-table-database-datasette

https://github.com/simonw/datasette/blob/035dc5e7b95142d4a700819a8cc4ff64aefe4efe/datasette/hookspecs.py#L62-L64

The hook is called in two places in the codebase - when rendering a table (pks variable is already in scope here):

https://github.com/simonw/datasette/blob/6373bb341457e5becfd5b67792ac2c8b9ed7c384/datasette/views/table.py#L897-L904

And when rendering an arbitrary query:

https://github.com/simonw/datasette/blob/6373bb341457e5becfd5b67792ac2c8b9ed7c384/datasette/views/database.py#L377-L384

Note that in that second one table is None (which is also called out in the documentation) - pks would be None here too.

None of the potential solutions for that problem are particularly appealing:

• Make these URL generation methods async - should the other ones be async too for consistency?
• Have some kind of mechanism that calculates and caches the pks for each table somewhere which can then be accessed here without an async call
• Require pks is passed to these methods, having been looked up elsewhere. This is a bit gross but may end up being the best solution

I'm considering changing these functions to accept the row object itself:

```python def row(self, database, table, row): ...

def row_blob(self, database, table, row, column): ... `` Just one catch: in order to generate the correctrow_pathwe need to know the primary keys for that table. We can look those up based on having access todatabaseandtable, but doing so requires anawait ...operation - and these functions are notasync`.

Also related: row is now available to render_cell() hook as-of this issue: - https://github.com/simonw/datasette/issues/1300

Learnt that the types in Sqlite-utils differ somewhat from those in Sqlite. I've changed my test to account for this difference and the test has passed successfully. I will submit a PR.

I tried this on Datasette Cloud and swapping datasetteproject/datasette to python:3.10-slim-bullseye as the base image dropped the overall image size from ~232MB to ~166MB - but note that I don't have SpatiaLite in that second image.

Yeah my initial implementation there makes no sense: python csv_reader_args = {"dialect": dialect} if delimiter: csv_reader_args["delimiter"] = delimiter if quotechar: csv_reader_args["quotechar"] = quotechar reader = _extra_key_strategy( csv_std.reader(decoded, **csv_reader_args), ignore_extras, extras_key ) first_row = next(reader) if no_headers: headers = ["untitled_{}".format(i + 1) for i in range(len(first_row))] reader = itertools.chain([first_row], reader) else: headers = first_row docs = (dict(zip(headers, row)) for row in reader) Because my _extra_key_strategy() helper function is designed to work against csv.DictReader - not against csv.reader() which returns a sequence of lists, not a sequence of dictionaries.

In fact, what's happening here is that dict(zip(headers, row)) is ignoring anything in the row that doesn't correspond to a header:

```pycon

list(zip(["a", "b"], [1, 2, 3])) [('a', 1), ('b', 2)] ```

Updated documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-row-value-column-table-database-datasette

Adding row to this plugin hook is a smart idea.

Relevant Twitter thread: https://twitter.com/simonw/status/1544710944462254080

Looks like python:3.10-slim-bullseye is a good option, see https://pythonspeed.com/articles/base-image-python-docker-images/

Based on https://github.com/python/cpython/blob/3.11/Lib/imghdr.py I'm tempted to say that if the file starts with b'\211PNG\r\n\032\n' then it's a PNG, if it starts with b'GIF8 then it's a GIF, anything else I assume an ICO.

https://www.w3.org/2005/10/howto-favicon suggests that it only needs to be able to identify PNG, GIF or ICO.

I could vendor the necessary parts of imghdr - it's pretty tiny: https://github.com/python/cpython/blob/3.11/Lib/imghdr.py

I could use the imghdr standard library module, but frustratingly it's marked as deprecated in Python 3.11! https://docs.python.org/3/library/imghdr.html

Annoyingly it looks like the standard library mimetypes module only uses filenames as the clue, it doesn't look at the bytes themselves.

I'm using that here: https://github.com/simonw/datasette/blob/9f1eb0d4eac483b953392157bd9fd6cc4df37de7/datasette/utils/asgi.py#L261-L277

https://pypi.org/project/python-magic/ can inspect files, but I don't want to add a whole new dependency just for this one feature.

The content type thing is a bit weird because usually I'd base that on the file extension, but here the favicon.ico file extension doesn't necessarily reflect if the image itself is a PNG or some other thing.

Loading it using the existing templating system would be better I think, since that way both custom installations AND plugins could influence the favicon in the same way that they influence the templates.

I think I want this to be a default feature, not a plugin.

Some options:

• Allow favicon path to be set in metadata.yml somehow, or via a --setting
• Teach Datasette to check for templates/favicon.ico and fall back to the default static/favicon.png if that file does not exist
• Document how to over-ride the favicon using the existing --static mechanism (if that's even possible)
• Outsource this to a new plugin

Note that I need a way to set the content_type correctly too.

I've written the code and test. Would you be able to advise how to compare table columns in a pytest function properly? Experiencing a challenge when comparing columns.

Test: python def test_duplicate(fresh_db): table = fresh_db.create_table( "table1", { "text_col": str, "float_col": float, "int_col": int, "bool_col": bool, "bytes_col": bytes, "datetime_col": datetime.datetime, }, ) dt = datetime.datetime.now() b = bytes('hello world', 'utf-8') data = {"text_col": "Cleo", "float_col": 3.14, "int_col": -2, "bool_col": True, "bytes_col": b, "datetime_col": str(dt)} table1 = fresh_db["table1"] row_id = table1.insert(data).last_rowid table1.duplicate('table2') table2 = fresh_db["table2"] assert data == table2.get(row_id) assert table1.columns == table2.columns # FAILS HERE

Result:

Failure is due to column types being named differently -- e.g. 'FLOAT' vs 'REAL', 'INTEGER' vs 'INT'. How should I go about comparing columns while accounting for equivalent types?

Or did I miss out something in my duplication code correctly? Here's how I did it: in db.py, I've added the following code: ```python class Table(Queryable): [...] def duplicate( self, name_new: str ) -> "Table": """ Duplicate this table in this database.

    :param name_new: Name of new table.
    """
    assert self.exists()
    with self.db.conn:
        sql = "CREATE TABLE [{new_table}] AS SELECT * FROM [{table}];".format(
            new_table = name_new,
            table = self.name,
        )
        self.db.execute(sql)
    return self.db[name_new]

```

This feature is pretty important and would be nice if it would be all within Datasette (no separate CLI/deploy required). My workflow now is to basically just copy the result and paste into a Google Sheet, which works, but then it's not discoverable to other journalists browsing the Datasette instance. I started building a plugin similar to datasette-saved-queries but one that maintains its own DB (required if you're working with all immutable DBs), but got bogged down in details.

Codecov Report

Merging #1766 (257fd24) into main (9f1eb0d) will increase coverage by 0.00%. The diff coverage is 100.00%.

```diff @@ Coverage Diff @@

main #1766 +/-

======================================= Coverage 91.67% 91.67%
======================================= Files 36 36
Lines 4658 4659 +1
======================================= + Hits 4270 4271 +1
Misses 388 388
```

| Impacted Files | Coverage Δ | | |---|---|---| | datasette/app.py | 94.27% <100.00%> (+<0.01%) | :arrow_up: |

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 9f1eb0d...257fd24. Read the comment docs.

Adding .sqlite3 to this feature makes sense to me.

I don't think it's worth having a --extensión option - it's possible right now to write a Datasette plugin that loads additional databases on startup, there's an example of how to do that in the source code for https://github.com/simonw/datasette-upload-dbs

Here: https://github.com/simonw/datasette-upload-dbs/blob/81510ef4f0004371a3adda18100936b430535cc1/datasette_upload_dbs/init.py#L41-L50

I'm affected by this as well. Would be nice to be able to pass in an extension, eg --extension=sqlite3.

This should work to avoid that:

sqlite-utils convert \
  ${dbfile} details dob \
  'r.parsedate(value)' \
  --where "dob != '*'"

I like the idea of this as a feature - design could look like this:

db["my_table"].duplicate("new_table")

Then for the CLI tool:

sqlite-utils duplicate data.db my_table new_table

Additional features that might be useful:

• Duplicate across connection aliases - so you can copy a table to another database using the duplicate method or command
• Ability to execute an arbitrary SQL query and save the results to a new table - basically a wrapper around CREATE TABLE t AS SELECT ..., again with alias support for copying to another database

Note, I do not get this issue using my Intel MacBook Pro =/

Environment markd@Marks-MBP metabase % python3 --version Python 3.9.13 markd@Marks-MBP metabase % sqlite3 --version 3.37.0 2021-12-09 01:34:53 9ff244ce0739f8ee52a3e9671adb4ee54c83c640b02e3f9d185fd2f9a179aapl markd@Marks-MBP metabase % sqlite-utils --version sqlite-utils, version 3.27 markd@Marks-MBP metabase % sqlite3 github.db SQLite version 3.37.0 2021-12-09 01:34:53 Enter ".help" for usage hints. sqlite> .dbconfig defensive off dqs_ddl on dqs_dml on enable_fkey off enable_qpsg off enable_trigger on enable_view on fts3_tokenizer off legacy_alter_table on legacy_file_format off load_extension off no_ckpt_on_close off reset_database off trigger_eqp off trusted_schema on writable_schema off

I just ran into what appears to be the same issue on a MacBook Pro, M1 Pro.

Environment: markd@Marks-MacBook-Pro metabase % python --version Python 3.8.9 markd@Marks-MacBook-Pro metabase % sqlite3 --version 3.37.0 2021-12-09 01:34:53 9ff244ce0739f8ee52a3e9671adb4ee54c83c640b02e3f9d185fd2f9a179aapl markd@Marks-MacBook-Pro metabase % sqlite-utils --version sqlite-utils, version 3.27 markd@Marks-MacBook-Pro metabase % sqlite3 gh.db SQLite version 3.37.0 2021-12-09 01:34:53 Enter ".help" for usage hints. sqlite> .dbconfig defensive off dqs_ddl on dqs_dml on enable_fkey off enable_qpsg off enable_trigger on enable_view on fts3_tokenizer off legacy_alter_table on legacy_file_format off load_extension off no_ckpt_on_close off reset_database off trigger_eqp off trusted_schema on writable_schema off

Error markd@Marks-MacBook-Pro metabase % github-to-sqlite repos gh.db a8cteam51 Traceback (most recent call last): File "/Users/markd/Library/Python/3.8/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/Users/markd/Library/Python/3.8/lib/python/site-packages/github_to_sqlite/cli.py", line 268, in repos utils.ensure_db_shape(db) File "/Users/markd/Library/Python/3.8/lib/python/site-packages/github_to_sqlite/utils.py", line 688, in ensure_db_shape ensure_foreign_keys(db) File "/Users/markd/Library/Python/3.8/lib/python/site-packages/github_to_sqlite/utils.py", line 682, in ensure_foreign_keys db[table].add_foreign_key(column, table2, column2) File "/Users/markd/Library/Python/3.8/lib/python/site-packages/sqlite_utils/db.py", line 2004, in add_foreign_key self.db.add_foreign_keys([(self.name, column, other_table, other_column)]) File "/Users/markd/Library/Python/3.8/lib/python/site-packages/sqlite_utils/db.py", line 1019, in add_foreign_keys cursor.execute( sqlite3.OperationalError: table sqlite_master may not be modified

Have you seen MakeTypes? Not the exact same thing but it may be relevant.

And it's inspired by the paper "Types from Data: Making Structured Data First-Class Citizens in F#".