```",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2005/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1552368054,I_kwDOBm6k_c5ch0G2,2000,rewrite_sql hook,193185,open,0,,,1,2023-01-23T01:02:52Z,2023-01-23T06:08:01Z,,CONTRIBUTOR,,"I'm not sold that this is a good idea, but thought it'd be worth writing up a ticket. Proposal: add a hook like
```python
def rewrite_sql(datasette, database, request, fn, sql, params)
```
It would be called from Database.execute, Database.execute_write, Database.execute_write_script, Database.execute_write_many before running the user's SQL. `fn` would indicate which method was being used, in case that's relevant for the SQL inspection -- for example `execute` only permits a single statement.
The hook could return a SQL statement to be executed instead, or an async function to be awaited on that returned the SQL to be executed.
Plugins that could be written with this hook:
- https://github.com/cldellow/datasette-ersatz-table-valued-functions would use this to avoid monkey-patching
- a plugin to inspect and reject unsafe Spatialite function calls (reported by [Simon in Discord](https://discord.com/channels/823971286308356157/823971286941302908/1066438832293159004))
- a plugin to do more general rewrites of queries to enforce table or row-level security, for example, based on the currently logged in actor's ID
- a plugin to maintain audit tables when users write to a table
- a plugin to cache expensive queries (eg the queries that drive facets) - these could allow stale reads if previously cached, then refresh them in an offline queue
Flaws with this idea:
`execute_fn` and `execute_write_fn` would not go through this hook, which limits the guarantees you can make about it for security purposes.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2000/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1560662739,I_kwDOBm6k_c5dBdLT,2007,`render_cell()` hook should take an optional `request` argument,9599,closed,0,,,1,2023-01-28T03:13:00Z,2023-08-09T17:15:03Z,2023-01-28T03:34:26Z,OWNER,,From Discord: https://discordapp.com/channels/823971286308356157/996877076982415491/1068227071156965486,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2007/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1563264257,I_kwDOBm6k_c5dLYUB,2010,Row page should default to card view,9599,open,0,,3268330,1,2023-01-30T21:49:37Z,2023-01-30T21:52:06Z,,OWNER,,"Datasette currently uses the same table layout on the row pages as it does on the table pages:
https://datasette.io/content/pypi_packages?_sort=name&name__exact=datasette-column-inspect
![]()
https://datasette.io/content/pypi_packages/datasette-column-inspect
![]()
If you shrink down to mobile width you get this instead, on both of those pages:
![]()
I think that view, which I think of as the ""card view"", is plain better if you're looking at just a single row - and it (or a variant of it) should be the default presentation on the row page.
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2010/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1564769997,I_kwDOBm6k_c5dRH7N,2011,"Applied facet did not result in an ""x"" icon to dismiss it",9599,open,0,,,1,2023-01-31T17:57:44Z,2023-01-31T17:58:54Z,,OWNER,,"
That's against this data https://data.sfgov.org/City-Management-and-Ethics/Supplier-Contracts/cqi5-hm2d imported using https://datasette.io/plugins/datasette-socrata
It's for `Contract Type` of `Non-Purchasing Contract (Rents, etc.)` - so possible that some of the spaces or punctuation in either the name of the value tripped up the code that decides if the X icon should be displayed.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2011/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1571207083,I_kwDOBm6k_c5dprer,2016,Database metadata fields like description are not available in the index page template's context,9993,open,0,,3268330,1,2023-02-05T02:25:53Z,2023-02-05T22:56:43Z,,NONE,,"When looping through `databases` in the index.html template, I'd like to print the description of each database alongside its name. But it appears that isn't passed in from the view, unless I'm missing it. It would be great to have that.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2016/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1578609658,I_kwDOBm6k_c5eF6v6,2022,Error 500 - not clear the cause,1667631,closed,0,,,1,2023-02-09T20:57:17Z,2023-02-09T21:13:50Z,2023-02-09T21:13:50Z,NONE,,"On the database that I have sent via linkedIn, datasette works great, but the following URL gives a 500 error.
http://127.0.0.1:8001/literature/authors_papers?authorId=100550354
The cause of the error is not apparent.
Is this expected behaviour?
David",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2022/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1579973223,I_kwDOBm6k_c5eLHpn,2024,Mention WAL mode in documentation,9599,open,0,,,1,2023-02-10T16:11:10Z,2023-02-10T16:11:53Z,,OWNER,,It's not currently obvious from the docs how you can ensure that Datasette runs well in situations where other processes may update the underlying SQLite files.,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2024/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1649791661,I_kwDOBm6k_c5iVdKt,2050,Row page JSON should use new ?_extra= format,9599,open,0,,8755003,1,2023-03-31T17:56:53Z,2023-03-31T17:59:49Z,,OWNER,,"https://latest.datasette.io/fixtures/facetable/2.json
Related:
- #2049
- #1709 ",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2050/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1665053646,I_kwDOBm6k_c5jPrPO,2059,"""Deceptive site ahead"" alert on Heroku deployment",1186275,open,0,,,1,2023-04-12T18:34:51Z,2023-04-13T01:13:01Z,,NONE,,"I deployed a fairly basic instance of Datasette (`datasette-auth-passwords` is the only plugin) using Heroku. The deployed URL now gives a ""Deceptive site ahead"" warning to users.
Is there way around this? Maybe a way to add ownership verification [through Google's search console](https://search.google.com/search-console/welcome)? ",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2059/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1690765434,I_kwDOBm6k_c5kxwh6,2067,Litestream-restored db: errors on 3.11 and 3.10.8; but works on py3.10.7 and 3.10.6,39538958,open,0,,,1,2023-05-01T12:42:28Z,2023-05-03T00:16:03Z,,NONE,,"Hi! Wondering if this issue is limited to my local system or if it affects others as well.
It seems like 3.11 errors out on a ""litestream-restored"" database. On further investigation, it also appears to conk out on 3.10.8 but works on 3.10.7 and 3.10.6.
To demo issue I created a test database, replicated it to an aws s3 bucket, then restored the same under various .pyenv-versioned shells where I test whether I can read the database via the sqlite3 cli.
```sh
# create new shell with 3.11.3
litestream restore -o data/db.sqlite s3://mytestbucketxx/db
sqlite3 data/db.sqlite
# SQLite version 3.41.2 2023-03-22 11:56:21
# Enter "".help"" for usage hints.
# sqlite> .tables
# _litestream_lock _litestream_seq movie
# sqlite>
```
However this get me an `OperationalError` when reading via datasette:
Error on 3.11.3 and 3.10.8
```sh
datasette data/db.sqlite
```
```console
/tester/.venv/lib/python3.11/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
warnings.warn(""pkg_resources is deprecated as an API"", DeprecationWarning)
Traceback (most recent call last):
File ""/tester/.venv/bin/datasette"", line 8, in
sys.exit(cli())
^^^^^
File ""/tester/.venv/lib/python3.11/site-packages/click/core.py"", line 1130, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/tester/.venv/lib/python3.11/site-packages/click/core.py"", line 1055, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File ""/tester/.venv/lib/python3.11/site-packages/click/core.py"", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/tester/.venv/lib/python3.11/site-packages/click/core.py"", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/tester/.venv/lib/python3.11/site-packages/click/core.py"", line 760, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/tester/.venv/lib/python3.11/site-packages/datasette/cli.py"", line 143, in wrapped
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File ""/tester/.venv/lib/python3.11/site-packages/datasette/cli.py"", line 615, in serve
asyncio.get_event_loop().run_until_complete(check_databases(ds))
File ""/Users/mv/.pyenv/versions/3.11.3/lib/python3.11/asyncio/base_events.py"", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File ""/tester/.venv/lib/python3.11/site-packages/datasette/cli.py"", line 660, in check_databases
await database.execute_fn(check_connection)
File ""/tester/.venv/lib/python3.11/site-packages/datasette/database.py"", line 213, in execute_fn
return await asyncio.get_event_loop().run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/Users/mv/.pyenv/versions/3.11.3/lib/python3.11/concurrent/futures/thread.py"", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/tester/.venv/lib/python3.11/site-packages/datasette/database.py"", line 211, in in_thread
return fn(conn)
^^^^^^^^
File ""/tester/.venv/lib/python3.11/site-packages/datasette/utils/__init__.py"", line 951, in check_connection
for r in conn.execute(
^^^^^^^^^^^^^
sqlite3.OperationalError: unable to open database file
```
Works on 3.10.7, 3.10.6
```sh
# create new shell with 3.10.7 / 3.10.6
litestream restore -o data/db.sqlite s3://mytestbucketxx/db
datasette data/db.sqlite
# ...
# INFO: Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
```
In both scenarios, the only dependencies were the pinned python version and the latest Datasette version 0.64.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2067/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1698865182,I_kwDOBm6k_c5lQqAe,2069,[BUG] Cannot insert new data to deployed instance,31861128,open,0,,,1,2023-05-07T02:59:42Z,2023-05-07T03:17:35Z,,NONE,,"## Summary
Recently, I deployed an instance of datasette to Vercel with the following plugins:
- datasette-auth-tokens
- datasette-insert
With the above plugins, I was able to insert new data to local sqlite db. However, when it comes to the deployment on Vercel, things behave differently. I observed some errors from the logs console on Vercel:
```console
File ""/var/task/datasette/database.py"", line 179, in _execute_writes
conn = self.connect(write=True)
File ""/var/task/datasette/database.py"", line 93, in connect
assert not (write and not self.is_mutable)
AssertionError
```
![]()
I think it is a potential bug.
## Reproduce
metadata.json
```json
{
""plugins"": {
""datasette-insert"": {
""allow"": {
""id"": ""*""
}
},
""datasette-auth-tokens"": {
""tokens"": [
{
""token"": {
""$env"": ""INSERT_TOKEN""
},
""actor"": {
""id"": ""repeater""
}
}
],
""param"": ""_auth_token""
}
}
}
```
commands
```bash
# deploy
datasette publish vercel remote.db \
--project=repeater-bot-sqlite \
--metadata metadata.json \
--install datasette-auth-tokens \
--install datasette-insert \
--vercel-json=vercel.json
# test insert
cat fixtures/dogs.json | curl --request POST -d @- -H ""Authorization: Bearer "" \
'https://repeater-bot-sqlite.vercel.app/-/insert/remote/dogs?pk=id'
```
logs
```console
Traceback (most recent call last):
File ""/var/task/datasette/app.py"", line 1354, in route_path
response = await view(request, send)
File ""/var/task/datasette/app.py"", line 1500, in async_view_fn
response = await async_call_with_supported_arguments(
File ""/var/task/datasette/utils/__init__.py"", line 1005, in async_call_with_supported_arguments
return await fn(*call_with)
File ""/var/task/datasette_insert/__init__.py"", line 14, in insert_or_upsert
response = await insert_or_upsert_implementation(request, datasette)
File ""/var/task/datasette_insert/__init__.py"", line 91, in insert_or_upsert_implementation
table_count = await db.execute_write_fn(write_in_thread, block=True)
File ""/var/task/datasette/database.py"", line 167, in execute_write_fn
raise result
File ""/var/task/datasette/database.py"", line 179, in _execute_writes
conn = self.connect(write=True)
File ""/var/task/datasette/database.py"", line 93, in connect
assert not (write and not self.is_mutable)
AssertionError
```
![]()
The `:p0` form field is missing. Submitting the form results in this error:
![]()
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2132/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1847201263,I_kwDOBm6k_c5uGg3v,2140,Remove all remaining documentation instances of '$ ',9599,closed,0,,,1,2023-08-11T17:42:13Z,2023-08-11T17:52:25Z,2023-08-11T17:45:00Z,OWNER,,"For example this: https://github.com/simonw/datasette/blob/4535568f2ce907af646304d0ebce2500ebd55677/docs/authentication.rst?plain=1#L33-L35
The problem with that `$ ` prefix is that it prevents users from copying and pasting the raw command.
![]()
https://docs.datasette.io/en/stable/authentication.html#using-the-root-actor",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2140/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1872043170,I_kwDOBm6k_c5vlRyi,2163,Rename core_X to catalog_X in the internals,9599,closed,0,,,1,2023-08-29T16:45:00Z,2023-08-29T17:01:31Z,2023-08-29T17:01:31Z,OWNER,,"Discussed with Alex this morning. We think the American spelling is fine here (it's shorter than `catalogue`) and that it's a slightly less lazy name than `core_`.
Follows:
- https://github.com/simonw/datasette/issues/2157",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2163/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1874255116,I_kwDOBm6k_c5vtt0M,2164,Ability to only load a specific list of plugins,9599,closed,0,,,1,2023-08-30T19:33:41Z,2023-09-08T04:35:46Z,2023-08-30T22:12:27Z,OWNER,,"I'm going to try and get this working through an environment variable, so that you can start Datasette and it will only load a subset of plugins including those that use the `register_commands()` hook.
Initial research on this:
- https://github.com/pytest-dev/pluggy/issues/422",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2164/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1884408624,I_kwDOBm6k_c5wUcsw,2177,Move schema tables from _internal to _catalog,9599,open,0,,,1,2023-09-06T16:58:33Z,2023-09-06T17:04:30Z,,OWNER,,"This came up in discussion over:
- https://github.com/simonw/datasette/pull/2174
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2177/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1817281557,I_kwDOC8SPRc5sUYQV,37,cannot use jinja filters in display?,10352819,closed,0,,,1,2023-07-23T20:09:54Z,2023-07-23T20:18:27Z,2023-07-23T20:18:26Z,NONE,,"Hi, I'm trying to have a display function in Dogsheep's `config.yml` that includes something like this:
```
{{ display.snippet|safe }}
```
Unfortunately, rendering fails with a message 'urls is undefined'.
The same happens if I'm trying to build a row URL manually, using filters like `quote_plus` (as my keys are URLs).
Any hints?
Thanks!",197431109,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/37/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1044267332,I_kwDOCGYnMM4-PkFE,336,"sqlite-util tranform --column-order mangles columns of type ""timestamp""",536941,closed,0,,,1,2021-11-04T01:15:38Z,2023-05-08T21:13:38Z,2023-05-08T21:13:38Z,CONTRIBUTOR,,"Reproducible code below:
```bash
> echo 'create table bar (baz text, created_at timestamp default CURRENT_TIMESTAMP)' | sqlite3 foo.db
> sqlite3 foo.db
SQLite version 3.36.0 2021-06-18 18:36:39
Enter "".help"" for usage hints.
sqlite> .schema bar
CREATE TABLE bar (baz text, created_at timestamp default CURRENT_TIMESTAMP);
sqlite> .exit
> sqlite-utils transform foo.db bar --column-order baz
sqlite3 foo.db
SQLite version 3.36.0 2021-06-18 18:36:39
Enter "".help"" for usage hints.
sqlite> .schema bar
CREATE TABLE IF NOT EXISTS ""bar"" (
[baz] TEXT,
[created_at] FLOAT DEFAULT 'CURRENT_TIMESTAMP'
);
sqlite> .exit
> sqlite-utils transform foo.db bar --column-order baz
> sqlite3 foo.db
SQLite version 3.36.0 2021-06-18 18:36:39
Enter "".help"" for usage hints.
sqlite> .schema bar
CREATE TABLE IF NOT EXISTS ""bar"" (
[baz] TEXT,
[created_at] FLOAT DEFAULT '''CURRENT_TIMESTAMP'''
);
```
",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/336/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1028056713,I_kwDOCGYnMM49RuaJ,332,`sqlite-utils memory --flatten` option to flatten nested JSON,22523840,closed,0,,,1,2021-10-16T14:04:42Z,2021-11-14T23:05:05Z,2021-11-14T23:05:05Z,NONE,,"currently --flatten option works only for `insert` command, it would be cool if it worked for `memory` as well to query nested json",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/332/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1072780607,I_kwDOCGYnMM4_8VU_,351,Support `--import xml.etree.ElementTree` in `sqlite-utils convert`,9599,closed,0,,,1,2021-12-07T00:40:29Z,2021-12-11T00:11:25Z,2021-12-11T00:11:25Z,OWNER,,"It's not possible to use a module that requires a nested import, such as `xml.etree.ElementTree`, at the moment. I found and fixed this bug in `git-history`, I should replicate that fix (and accompanying documentation) here: https://github.com/simonw/git-history/issues/39",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/351/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1976986318,I_kwDOCGYnMM511mrO,599,Cannot find spatialite on arm64 linux,37802088,closed,0,,,1,2023-11-03T22:05:51Z,2023-11-04T01:06:31Z,2023-11-04T00:33:28Z,CONTRIBUTOR,,"Initially, I found an issue in `datasette` where it wouldn’t find `spatialite` when running on my Radxa Rock 5B - an RK3588 powered SBC, running the arm64 build of Debian Bullseye. I confirmed the same behaviour on my Raspberry Pi 4 - a BCM2711 powered SBC, running the arm64 build of Debian Bookworm.
```
$ datasette --load-extension=spatialite example.db
Error: Could not find SpatiaLite extension
```
I did some digging and realised the issue originates in this project. Even with the `libsqlite3-mod-spatialite` package installed, `pytest` skips all of the GIS tests in the project.
```
$ apt list --installed | grep spatial
[…]
libsqlite3-mod-spatialite/stable,now 5.0.1-3 arm64 [installed]
$ ls -l /usr/lib/*/*spatial*
lrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so -> mod_spatialite.so.7.1.0
lrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7 -> mod_spatialite.so.7.1.0
-rw-r--r-- 1 root root 7348584 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7.1.0
```
```
$ pytest
tests/test_get.py ...... [ 73%]
tests/test_gis.py ssssssssssss [ 75%]
tests/test_hypothesis.py .... [ 75%]
```
I tracked the issue down to the [`find_sqlite()` function in the `utils.py`](https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/utils.py#L60) file. The [`SPATIALITE_PATHS`](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/utils.py#L34-L39) array doesn’t have an entry for the location of this module on arm64 linux.
",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/599/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1988525411,I_kwDOCGYnMM52hn1j,603,Pyhton 3.12 Bug report,1324252,open,0,,,1,2023-11-10T22:57:48Z,2023-12-08T05:10:31Z,,NONE,,"I start with new python3 verison 3.12.0
Also have the error where connect DataBase
```
Traceback (most recent call last):
File ""/home/t/Development/python/FKPJ/ClinicSYS/run.py"", line 1, in
import re, os, io, json, sqlite_utils, requests, pytz, logging
File ""/home/t/.local/lib/python3.12/site-packages/sqlite_utils/__init__.py"", line 1, in
from .db import Database
File ""/home/t/.local/lib/python3.12/site-packages/sqlite_utils/db.py"", line 277, in
class Database:
File ""/home/t/.local/lib/python3.12/site-packages/sqlite_utils/db.py"", line 306, in Database
filename_or_conn: Optional[Union[str, pathlib.Path, sqlite3.Connection]] = None,
^^^^^^^^^^^^^^^^^^
```
This bug come from `sqlite-utils` since's v3.33.
Anyone get the same ?
As well now of the resolved plan just keep the sqlite-utils version in python3.12 with v3.32.1 [tested]
but where are the sqlite3.Connection problem....
This won't happen on python version down to 3.11[tested]
Just the python3.12.0, I have test this error are come from the sqlite3 connection
The error say from `sqlite_utils` and with the sqlite3 Connection, what can I do.
Let fix together.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/603/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
2007893839,I_kwDOCGYnMM53rgdP,605,Insert fails with `Error: Python int too large to convert to SQLite INTEGER`; can we use `NUMERIC` here?,12229877,closed,0,,,1,2023-11-23T10:19:46Z,2023-12-08T05:07:54Z,2023-12-08T05:07:54Z,NONE,,"I'm currently working on a new feature for Hypothesis, where we can dump a tidy jsonlines table of all the test cases we tried - including arguments, outcomes, timings, coverage, etc. Exploring this seems like a perfect cases for `sqlite-utils` and `datasette`, but I pretty quickly ran into an integer overflow problem and don't want to recommend that experience to my users.
I originally went to report this as a bug... and then found https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895581038 almost exactly matched my repro 😅
https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391063 suggests that using `NUMERIC` would avoid this overflow error, although ""If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL."" suggests that this would come at the cost of rounding to the nearest float value. Maybe I should just convert large integers to float before writing out my json?
After a bit more hacking, ""manually cast large integers to float"" seems like a decent solution for my particular case, but having written it up I thought I might as well post this issue anyway - I hope it's useful feedback, and won't mind at all if you close as wontfix if it's not.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/605/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1079422215,I_kwDOCGYnMM5AVq0H,357,pytest-runner is not required,4067843,closed,0,,,1,2021-12-14T07:51:24Z,2021-12-16T20:43:19Z,2021-12-16T20:43:13Z,NONE,,Deprecated pytest-runner is not necessary for running the testsuite.,140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/357/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1091819089,I_kwDOCGYnMM5BE9ZR,360,MemoryError,559453,closed,0,,,1,2022-01-01T13:39:17Z,2022-03-21T04:22:46Z,2022-03-21T04:22:46Z,NONE,,"HI, when dealing with large json file (~170GB) i got the following error
```
Traceback (most recent call last):
File ""/usr/local/bin/sqlite-utils"", line 8, in
sys.exit(cli())
File ""/usr/lib/python3/dist-packages/click/core.py"", line 1126, in __call__
return self.main(*args, **kwargs)
File ""/usr/lib/python3/dist-packages/click/core.py"", line 1051, in main
rv = self.invoke(ctx)
File ""/usr/lib/python3/dist-packages/click/core.py"", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/usr/lib/python3/dist-packages/click/core.py"", line 1393, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/usr/lib/python3/dist-packages/click/core.py"", line 752, in invoke
return __callback(*args, **kwargs)
File ""/usr/local/lib/python3.9/dist-packages/sqlite_utils/cli.py"", line 1300, in memory
rows, format_used = rows_from_file(csv_fp, format=format, encoding=encoding)
File ""/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py"", line 185, in rows_from_file
return rows_from_file(buffered, format=Format.JSON)
File ""/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py"", line 156, in rows_from_file
decoded = json.load(fp)
File ""/usr/lib/python3.9/json/__init__.py"", line 293, in load
return loads(fp.read(),
MemoryError
```",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/360/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1097129710,I_kwDOCGYnMM5BZN7u,372,Idea: `suffix` and `stem` file columns,9599,closed,0,,7558727,1,2022-01-09T07:48:53Z,2022-01-10T19:27:34Z,2022-01-09T20:17:00Z,OWNER,,"For https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files
Given a file called `dogs.jpg` stem would be `dogs` and ext would be `jpg`.
Need to decide what happens for `dogs.and.cats.jpg.gz`.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/372/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1098574572,I_kwDOCGYnMM5Beurs,380,Release notes for 3.21,9599,closed,0,,7558727,1,2022-01-11T02:12:30Z,2022-01-11T02:34:26Z,2022-01-11T02:34:26Z,OWNER,,For these commits: https://github.com/simonw/sqlite-utils/compare/3.20...129141572f249ea290e2a075437e2ebaad215859,140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/380/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1099585611,I_kwDOCGYnMM5BilhL,382,`--where` option for `sqlite-rows`,9599,closed,0,,,1,2022-01-11T20:24:23Z,2022-01-11T23:33:14Z,2022-01-11T23:32:47Z,OWNER,,CLI equivalent of `table.rows_where()` - should accept parameters too. Work on this at the same time as #381.,140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/382/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1122446693,I_kwDOCGYnMM5C5y1l,394,Test against Python 3.11-dev,9599,open,0,,,1,2022-02-02T22:21:03Z,2022-02-03T21:06:35Z,,OWNER,,"Same as:
- https://github.com/simonw/datasette/issues/1621",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/394/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1111293050,I_kwDOCGYnMM5CPPx6,387,Python library docs should start with a self contained example,9599,closed,0,,,1,2022-01-22T06:23:56Z,2022-01-26T01:37:17Z,2022-01-26T01:35:30Z,OWNER,,You have to read a lot of stuff in a lot of different places to get started with the Python library. Add a getting started introduction to https://sqlite-utils.datasette.io/en/stable/python-api.html,140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/387/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1123849278,I_kwDOCGYnMM5C_JQ-,395,"""apt-get: command not found"" error on macOS",9599,closed,0,,,1,2022-02-04T06:03:42Z,2022-02-04T06:10:58Z,2022-02-04T06:10:58Z,OWNER,,"Yeah, `apt-get` isn't a thing on macOS so 4a2a3e2fd0d5534f446b3f1fee34cb165e4d86d2 (to test #79 against real SpatiaLite) broke.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/395/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1114557284,I_kwDOCGYnMM5Cbstk,390,`sqlite-utils upsert` should require `--pk` more elegantly,9599,closed,0,,,1,2022-01-26T02:20:31Z,2022-01-26T03:20:25Z,2022-01-26T03:19:43Z,OWNER,,"Currently throws an ugly traceback:
```
% echo '[
{""id"": 1, ""name"": ""Lila""},
{""id"": 1, ""name"": ""Lila""}
]' | sqlite-utils upsert data.db chickens -
Traceback (most recent call last):
File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils"", line 33, in
sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())
File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1137, in __call__
return self.main(*args, **kwargs)
File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1062, in main
rv = self.invoke(ctx)
File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1668, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 763, in invoke
return __callback(*args, **kwargs)
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 1104, in upsert
insert_upsert_implementation(
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 906, in insert_upsert_implementation
db[table].insert_all(
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2615, in insert_all
raise PrimaryKeyRequired(""upsert() requires a pk"")
sqlite_utils.db.PrimaryKeyRequired: upsert() requires a pk
```",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/390/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1118585417,I_kwDOCGYnMM5CrEJJ,393,Better documentation for insert-replace,9599,closed,0,,,1,2022-01-30T15:40:23Z,2022-02-03T22:13:24Z,2022-02-03T22:13:24Z,OWNER,,"Currently: https://sqlite-utils.datasette.io/en/stable/python-api.html#insert-replacing-data
> If you want to insert a record or replace an existing record with the same primary key, using the replace=True argument to .insert() or .insert_all():
Should describe the exception you get first, then how to use replace to avoid it.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/393/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1125077063,I_kwDOCGYnMM5DD1BH,400,`sqlite-utils create-table` ... `--if-not-exists`,9599,closed,0,,,1,2022-02-06T01:32:53Z,2022-02-06T01:34:53Z,2022-02-06T01:34:46Z,OWNER,,"Inspired by:
- #397
To match the option on `create-index`: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-index
```
--if-not-exists Ignore if index already exists
```",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/400/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1181236173,I_kwDOCGYnMM5GaDvN,422,Reconsider not running convert functions against null values,9599,open,0,,,1,2022-03-25T20:22:40Z,2022-03-25T20:23:21Z,,OWNER,,"I just got caught out by the fact that `None` values are not processed by the `.convert()` mechanism https://github.com/simonw/sqlite-utils/blob/0b7b80bd40fe86e4d66a04c9f607d94991c45c0b/sqlite_utils/db.py#L2504-L2510
I had run this code while working on #420 and I wasn't sure why it didn't work:
```
$ sqlite-utils add-column content.db articles score float
$ sqlite-utils convert content.db articles score '
import random
random.seed(10)
def convert(value):
global random
return random.random()
'
```
The reason it didn't work is that the newly added `score` column was full of `null` values.
I fixed it by doing this instead:
$ sqlite-utils add-column content.db articles score float --not-null-default 1.0
But this indicates to me that the design of `convert()` here may be incorrect.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/422/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,