In [10]: list(db[""type""].rows)
Out[10]: [{'id': 1, 'type': None}, {'id': 2, 'type': 'dog'}]
```",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/186/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
728905098,MDU6SXNzdWU3Mjg5MDUwOTg=,1048,Documentation and unit tests for urls.row() urls.row_blob() methods,9599,open,0,,,7,2020-10-25T00:13:53Z,2022-07-10T16:23:57Z,,OWNER,,https://github.com/simonw/datasette/blob/5db7ae3ce165ded57c7fb1cfbdb3258b1cf06c10/datasette/app.py#L1307-L1313,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1048/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
734777631,MDU6SXNzdWU3MzQ3Nzc2MzE=,1080,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",9599,open,0,,3268330,7,2020-11-02T19:55:06Z,2022-02-04T06:25:18Z,,OWNER,,Can use `/database/-/...` namespace from #296,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1080/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
743384829,MDExOlB1bGxSZXF1ZXN0NTIxMjg3OTk0,203,changes to allow for compound foreign keys,1049910,open,0,,,7,2020-11-16T00:30:10Z,2023-01-25T18:47:18Z,,FIRST_TIME_CONTRIBUTOR,simonw/sqlite-utils/pulls/203,"Add support for compound foreign keys, as per issue #117
Not sure if this is the right approach. In particular I'm unsure about:
- the new `ForeignKey` class, which replaces the namedtuple in order to ensure that `column` and `other_column` are forced into tuples. The class does the job, but doesn't feel very elegant.
- I haven't rewritten `guess_foreign_table` to take account of multiple columns, so it just checks for the first column in the foreign key definition. This isn't ideal.
- I haven't added any ability to the CLI to add compound foreign keys, it's only in the python API at the moment.
The PR also contains a minor related change that columns and tables are always quoted in foreign key definitions.",140912432,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/203/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
771511344,MDExOlB1bGxSZXF1ZXN0NTQzMDE1ODI1,31,Update for Big Sur,41546558,open,0,,,7,2020-12-20T04:36:45Z,2023-08-08T15:52:52Z,,CONTRIBUTOR,dogsheep/dogsheep-photos/pulls/31,Refactored out the SQL for extracting aesthetic scores to use osxphotos -- adds compatbility for Big Sur via osxphotos which has been updated for new table names in Big Sur. Have not yet refactored the SQL for extracting labels which is still compatible with Big Sur.,256834907,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
954546309,MDExOlB1bGxSZXF1ZXN0Njk4NDIzNjY3,8,Add Gmail takeout mbox import (v2),28565,open,0,,,7,2021-07-28T07:05:32Z,2023-09-08T01:22:49Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/google-takeout-to-sqlite/pulls/8,"WIP
This PR builds on #5 to continue implementing gmail import support.
Building on @UtahDave's work, these commits add a few performance and bug fixes:
* Decreased memory overhead for import by manually parsing mbox headers.
* Fixed error where some messages in the mbox would yield a row with NULL in all columns.
I will send more commits to fix any errors I encounter as I run the importer on my personal takeout data.",206649770,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
972918533,MDU6SXNzdWU5NzI5MTg1MzM=,1438,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions,9599,open,0,,,7,2021-08-17T17:35:36Z,2021-08-18T00:22:23Z,,OWNER,,"> Confirmed: https://thesession.vercel.app/thesession?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A is a page where the URL correctly encoded `%` as `%25` - but then in the HTML on that page that links to the CSV and JSON versions we get this:
>
> ```html
> This data as
> json,
> CSV
>
> ```
Those CSV and JSON links are incorrect.
_Originally posted by @simonw in https://github.com/simonw/datasette-publish-vercel/issues/48#issuecomment-900497579_",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1438/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1010112818,I_kwDOBm6k_c48NRky,1479,"Win32 ""used by another process"" error with datasette publish",76450761,open,0,,,7,2021-09-28T19:12:00Z,2023-09-07T02:14:16Z,,NONE,,"I unfortunately was not successful to deploy to fly.io. Please see the details above of the three scenarios that I took. I am also new to datasette.
Failed to deploy. Attaching logs:
1. Tried with an app created via `flyctl apps create frosty-fog-8565` and the ran `datasette publish fly covid.db --app frosty-fog-8565`
```
Deploying frosty-fog-8565
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
Error error connecting to docker: An unknown error occured.
Traceback (most recent call last):
File ""c:\users\grott\anaconda3\lib\runpy.py"", line 193, in _run_module_as_main
""__main__"", mod_spec)
File ""c:\users\grott\anaconda3\lib\runpy.py"", line 85, in _run_code
exec(code, run_globals)
File ""C:\Users\grott\Anaconda3\Scripts\datasette.exe\__main__.py"", line 7, in
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 782, in main
rv = self.invoke(ctx)
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""c:\users\grott\anaconda3\lib\site-packages\datasette_publish_fly\__init__.py"", line 156, in fly
""--remote-only"",
File ""c:\users\grott\anaconda3\lib\contextlib.py"", line 119, in __exit__
next(self.gen)
File ""c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py"", line 451, in temporary_docker_directory
tmp.cleanup()
File ""c:\users\grott\anaconda3\lib\tempfile.py"", line 811, in cleanup
_shutil.rmtree(self.name)
File ""c:\users\grott\anaconda3\lib\shutil.py"", line 516, in rmtree
return _rmtree_unsafe(path, onerror)
File ""c:\users\grott\anaconda3\lib\shutil.py"", line 395, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File ""c:\users\grott\anaconda3\lib\shutil.py"", line 404, in _rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File ""c:\users\grott\anaconda3\lib\shutil.py"", line 402, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\grott\\AppData\\Local\\Temp\\tmpgcm8cz66\\frosty-fog-8565'
```
2. Tried also with an app that gets autogenerate when running `flyctl launch`. This also generates the .toml file. Ran then `datasette publish fly covid.db --app dark-feather-168` **but different error now**
```Deploying dark-feather-168
==> Validating app configuration
Error not possible to validate configuration: server returned Post ""https://api.fly.io/graphql"": unexpected EOF
Traceback (most recent call last):
File ""c:\users\grott\anaconda3\lib\runpy.py"", line 193, in _run_module_as_main
""__main__"", mod_spec)
exec(code, run_globals)
File ""C:\Users\grott\Anaconda3\Scripts\datasette.exe\__main__.py"", line 7, in
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 782, in main
rv = self.invoke(ctx)
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""c:\users\grott\anaconda3\lib\site-packages\datasette_publish_fly\__init__.py"", line 156, in fly
""--remote-only"",
File ""c:\users\grott\anaconda3\lib\contextlib.py"", line 119, in __exit__
next(self.gen)
File ""c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py"", line 451, in temporary_docker_directory
tmp.cleanup()
File ""c:\users\grott\anaconda3\lib\tempfile.py"", line 811, in cleanup
_shutil.rmtree(self.name)
File ""c:\users\grott\anaconda3\lib\shutil.py"", line 516, in rmtree
return _rmtree_unsafe(path, onerror)
File ""c:\users\grott\anaconda3\lib\shutil.py"", line 395, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File ""c:\users\grott\anaconda3\lib\shutil.py"", line 404, in _rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File ""c:\users\grott\anaconda3\lib\shutil.py"", line 402, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\grott\\AppData\\Local\\Temp\\tmpnoyewcre\\dark-feather-168'
```
These are also the contents of the generated **.toml file** in 2 scenario:
```
# fly.toml file generated for dark-feather-168 on 2021-09-28T20:35:44+02:00
app = ""dark-feather-168""
kill_signal = ""SIGINT""
kill_timeout = 5
processes = []
[env]
[experimental]
allowed_public_ports = []
auto_rollback = true
[[services]]
http_checks = []
internal_port = 8080
processes = [""app""]
protocol = ""tcp""
script_checks = []
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = ""connections""
[[services.ports]]
handlers = [""http""]
port = 80
[[services.ports]]
handlers = [""tls"", ""http""]
port = 443
[[services.tcp_checks]]
grace_period = ""1s""
interval = ""15s""
restart_limit = 6
timeout = ""2s""
```
3. But also trying `datasette package covid.db` to create a local DOCKERFILE to later try to push it via `flyctl deploy` fails as well.
```[+] Building 147.3s (11/11) FINISHED
=> [internal] load build definition from Dockerfile 0.2s
=> => transferring dockerfile: 396B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3.8 4.7s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 82.37kB 0.0s
=> [1/5] FROM docker.io/library/python:3.8@sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3 108.3s
=> => resolve docker.io/library/python:3.8@sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3b5 0.0s
=> => sha256:56182bcdf4d4283aa1f46944b4ef7ac881e28b4d5526720a4e9ba03a4730846a 2.22kB / 2.22kB 0.0s
=> => sha256:955615a668ce169f8a1443fc6b6e6215f43fe0babfb4790712a2d3171f34d366 54.93MB / 54.93MB 21.6s
=> => sha256:911ea9f2bd51e53a455297e0631e18a72a86d7e2c8e1807176e80f991bde5d64 10.87MB / 10.87MB 15.5s
=> => sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3b5c5b6 1.86kB / 1.86kB 0.0s
=> => sha256:ff08f08727e50193dcf499afc30594c47e70cc96f6fcfd1a01240524624264d0 8.65kB / 8.65kB 0.0s
=> => sha256:2756ef5f69a5190f4308619e0f446d95f5515eef4a814dbad0bcebbbbc7b25a8 5.15MB / 5.15MB 6.4s
=> => sha256:27b0a22ee906271a6ce9ddd1754fdd7d3b59078e0b57b6cc054c7ed7ac301587 54.57MB / 54.57MB 37.7s
=> => sha256:8584d51a9262f9a3a436dea09ba40fa50f85802018f9bd299eee1bf538481077 196.45MB / 196.45MB 82.3s
=> => sha256:524774b7d3638702fe9ae0ea3fcfb81b027dfd75cc2fc14f0119e764b9543d58 6.29MB / 6.29MB 26.6s
=> => extracting sha256:955615a668ce169f8a1443fc6b6e6215f43fe0babfb4790712a2d3171f34d366 5.4s
=> => sha256:9460f6b75036e38367e2f27bb15e85777c5d6cd52ad168741c9566186415aa26 16.81MB / 16.81MB 40.5s
=> => extracting sha256:2756ef5f69a5190f4308619e0f446d95f5515eef4a814dbad0bcebbbbc7b25a8 0.6s
=> => extracting sha256:911ea9f2bd51e53a455297e0631e18a72a86d7e2c8e1807176e80f991bde5d64 0.6s
=> => sha256:9bc548096c181514aa1253966a330134d939496027f92f57ab376cd236eb280b 232B / 232B 40.1s
=> => extracting sha256:27b0a22ee906271a6ce9ddd1754fdd7d3b59078e0b57b6cc054c7ed7ac301587 5.8s
=> => sha256:1d87379b86b89fd3b8bb1621128f00c8f962756e6aaaed264ec38db733273543 2.35MB / 2.35MB 41.8s
=> => extracting sha256:8584d51a9262f9a3a436dea09ba40fa50f85802018f9bd299eee1bf538481077 18.8s
=> => extracting sha256:524774b7d3638702fe9ae0ea3fcfb81b027dfd75cc2fc14f0119e764b9543d58 1.2s
=> => extracting sha256:9460f6b75036e38367e2f27bb15e85777c5d6cd52ad168741c9566186415aa26 2.9s
=> => extracting sha256:9bc548096c181514aa1253966a330134d939496027f92f57ab376cd236eb280b 0.0s
=> => extracting sha256:1d87379b86b89fd3b8bb1621128f00c8f962756e6aaaed264ec38db733273543 0.8s
=> [2/5] COPY . /app 2.3s
=> [3/5] WORKDIR /app 0.2s
=> [4/5] RUN pip install -U datasette 26.9s
=> [5/5] RUN datasette inspect covid.db --inspect-file inspect-data.json 3.1s
=> exporting to image 1.2s
=> => exporting layers 1.2s
=> => writing image sha256:b5db0c205cd3454c21fbb00ecf6043f261540bcf91c2dfc36d418f1a23a75d7a 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Traceback (most recent call last):
""__main__"", mod_spec)
File ""c:\users\grott\anaconda3\lib\runpy.py"", line 85, in _run_code
exec(code, run_globals)
File ""C:\Users\grott\Anaconda3\Scripts\datasette.exe\__main__.py"", line 7, in
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 782, in main
rv = self.invoke(ctx)
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""c:\users\grott\anaconda3\lib\site-packages\datasette\cli.py"", line 283, in package
call(args)
File ""c:\users\grott\anaconda3\lib\contextlib.py"", line 119, in __exit__
next(self.gen)
File ""c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py"", line 451, in temporary_docker_directory
tmp.cleanup()
File ""c:\users\grott\anaconda3\lib\tempfile.py"", line 811, in cleanup
_shutil.rmtree(self.name)
File ""c:\users\grott\anaconda3\lib\shutil.py"", line 516, in rmtree
return _rmtree_unsafe(path, onerror)
File ""c:\users\grott\anaconda3\lib\shutil.py"", line 395, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File ""c:\users\grott\anaconda3\lib\shutil.py"", line 404, in _rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File ""c:\users\grott\anaconda3\lib\shutil.py"", line 402, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\grott\\AppData\\Local\\Temp\\tmpkb27qid3\\datasette'```",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1479/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1082651698,I_kwDOCGYnMM5Ah_Qy,358,Support for CHECK constraints,11597658,open,0,,,7,2021-12-16T21:19:45Z,2022-09-25T07:15:59Z,,NONE,,"Hi,
I noticed the `transform.table()` method doesn't have an option to add/change or drop a check constraint (see https://sqlite.org/lang_createtable.html -> 3.7 Check Constraints. would be great to have this as an option!
",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/358/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1100015398,I_kwDOBm6k_c5BkOcm,1591,Maybe let plugins define custom serve options?,9599,open,0,,,7,2022-01-12T08:18:47Z,2022-01-15T11:56:59Z,,OWNER,,"https://twitter.com/psychemedia/status/1481171650934714370
> can extensions be passed their own cli args? eg `--ext-tiddlywiki-dbname tiddlywiki2.sqlite` ?
I've thought something like this might be useful for other plugins in the past, too.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1591/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1363552780,I_kwDOBm6k_c5RRioM,1805,truncate_cells_html does not work for links?,562352,open,0,,,7,2022-09-06T16:41:29Z,2022-10-03T09:18:06Z,,NONE,,"We have many links inside our dataset (please don't blame us ;-).
When I use `--settings truncate_cells_html 60` it is not working for the links.
Eg. https://images.openfoodfacts.org/images/products/000/000/000/088/nutrition_fr.5.200.jpg (87 chars) is not truncated:
IMHO It would make sense that links should be treated as HTML. The link should work of course, but Datasette could truncate it:
[https://images.openfoodfacts.org/images/products/00[...].jpg](https://images.openfoodfacts.org/images/products/000/000/000/088/nutrition_fr.5.200.jpg)
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1805/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,reopened
1383646615,I_kwDOCGYnMM5SeMWX,491,Ability to merge databases and tables,8904453,open,0,,,7,2022-09-23T11:10:55Z,2023-06-14T22:14:24Z,,NONE,,"Hi! Let me firstly say that I am a big fan of your work -- I follow your tweets and blog posts with great interest 😄.
Now onto the matter at hand: I think it would be great if `sqlite-utils` included a `merge` or `combine` command, with the purpose of combining different SQLite databases into a single SQLite database. This way, the newly ""merged"" database would contain all differently named tables contained in the databases to be merged as-is, as well a concatenation of all tables of the same name.
This could look something like this:
```bash
sqlite-utils merge cats.db dogs.db > animals.db
```
I imagine this is rather straightforward if all databases involved in the merge contain differently named tables (i.e. no chance of conflicts), but things get slightly more complicated if two or more of the databases to be merged contain tables with the same name. Not only do you have to ""do something"" with the primary key(s), but these tables could also simply have different schemas (and therefore be incompatible for concatenation to begin with).
Anyhow, I would love your thoughts on this, and, if you are open to it, work together on the design and implementation!",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/491/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1384273985,I_kwDOBm6k_c5SglhB,1817,Expose `sql` and `params` arguments to various plugin hooks,9599,open,0,,,7,2022-09-23T20:34:45Z,2022-09-27T00:27:53Z,,OWNER,,"On Discord: https://discord.com/channels/823971286308356157/996877076982415491/1022784534363787305
> Hi! I'm attempting to write a plugin that would provide some statistics on text fields (most common words, etc). I would want this information displayed in the table pages, and (ideally) also updated when users make custom queries from the table pages.
>
> It seems one way to do this would be to use the extra_template_vars hook, and make the appropriate SQL query there. So extra_template_vars would create a variable that is a list of most common words, and this is displayed on the page, possibly above the regular table view.
>
> Is there a way that the plugin code can access the SQL query (or even the data) that was used to produce the table view? I can see that TableView class constructs the SQL query, but I can't seem to find a way to access that information from the objects that are available to extra_template_vars.
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1817/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1426379903,PR_kwDOBm6k_c5BtJNn,1870,"don't use immutable=1, only mode=ro",536941,open,0,,,7,2022-10-27T23:33:04Z,2023-10-03T19:12:37Z,,CONTRIBUTOR,simonw/datasette/pulls/1870,"Opening db files in immutable mode sometimes leads to the file being mutated, which causes duplication in the docker image layers: see #1836, #1480
That this happens in ""immutable"" mode is surprising, because the sqlite docs say that setting this should open the database as read only.
https://www.sqlite.org/c3ref/open.html
> immutable: The immutable parameter is a boolean query parameter that indicates that the database file is stored on read-only media. When immutable is set, SQLite assumes that the database file cannot be changed, even by a process with higher privilege, and so the database is opened read-only and all locking and change detection is disabled. Caution: Setting the immutable property on a database file that does in fact change can result in incorrect query results and/or [SQLITE_CORRUPT](https://www.sqlite.org/rescode.html#corrupt) errors. See also: [SQLITE_IOCAP_IMMUTABLE](https://www.sqlite.org/c3ref/c_iocap_atomic.html).
Perhaps this is a bug in sqlite?
----
:books: Documentation preview :books:: https://datasette--1870.org.readthedocs.build/en/1870/
",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1870/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1430797211,I_kwDOBm6k_c5VSDub,1875,Figure out design for JSON errors (consider RFC 7807),9599,open,0,,8755003,7,2022-11-01T03:14:15Z,2022-12-13T05:29:08Z,,OWNER,,"https://datatracker.ietf.org/doc/draft-ietf-httpapi-rfc7807bis/ is a brand new standard.
Since I need a neat, predictable format for my JSON errors, maybe I should use this one?",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1875/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1940346034,I_kwDOBm6k_c5zp1Sy,2199,Detailed upgrade instructions for metadata.yaml -> datasette.yaml,9599,open,0,,3268330,7,2023-10-12T16:21:25Z,2023-10-12T22:08:42Z,,OWNER,,"> `Exception: Datasette no longer accepts plugin configuration in --metadata. Move your ""plugins"" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.`
>
> I think we should link directly to documentation that tells people how to perform this upgrade.
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/2190#issuecomment-1759947021_
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2199/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
348043884,MDU6SXNzdWUzNDgwNDM4ODQ=,357,Plugin hook for loading metadata.json,9599,open,0,,,6,2018-08-06T19:00:01Z,2020-06-21T22:19:58Z,,OWNER,,"For https://github.com/simonw/russian-ira-facebook-ads-datasette/tree/af6d956995e14afd585c35a6a06bb01da32043ba I wrote a script to convert YAML to JSON because YAML is a better format for embedding multi-line HTML descriptions and canned SQL statements.
Example yaml metadata file: https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/af6d956995e14afd585c35a6a06bb01da32043ba/russian-ads-metadata.yaml
It would be useful if Datasette could be fed a YAML file directly:
datasette -m metadata.yaml
Question is... should this be a native feature (hence adding a YAML dependency) or should it be handled by a `datasette-metadata-yaml` plugin, using a new plugin hook for loading metadata? If so, what would other use-cases for that plugin hook be?",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/357/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
520667773,MDU6SXNzdWU1MjA2Njc3NzM=,620,Mechanism for indicating foreign key relationships in the table and query page URLs,9599,open,0,,,6,2019-11-10T22:26:27Z,2021-04-05T03:57:22Z,,OWNER,,"Datasette currently only inflates foreign keys (into names hyperlinks) if it detects them as foreign key constraints in the underlying database.
It would be useful if you could specify additional ""foreign keys"" using both `metadata.json` and the querystring - similar time how you can pass `?_fts_table=x` https://datasette.readthedocs.io/en/stable/full_text_search.html#configuring-full-text-search-for-a-table-or-view",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/620/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",,
527670799,MDU6SXNzdWU1Mjc2NzA3OTk=,639,updating metadata.json without recreating the app,172847,open,0,,,6,2019-11-24T09:19:53Z,2019-11-30T06:08:50Z,,NONE,,"I've sucessfully ""uploaded"" an SQLite database (with a metadata.json file) to heroku using:
$ datasette publish heroku so-sales.db -m metadata.json -n so-sales
The question is: how can I modify the (small) metadata.json file without having to upload the (large) SQLite database.
The directions on heroku indicate I should run:
heroku git:clone -a so-sales
But this just results in an empty directory with a warning:
warning: You appear to have cloned an empty repository.
I've been able to ""clone"" the heroku ""app"" using the command:
$ heroku slugs:download -a so-sales
but this is not a git repository....
Ideally, it seems to me, there'd be an option of the `datasette` CLI to allow a file
to be updated, or there'd be some way to create a local git ""clone"" of the app
so that the heroku instructions for ""Deploying with git"" would apply.
(p.s. I ran `datasette publish heroku -m metadata.json -n so-sales`
in the hope that that would not cause the .db file to be wiped, but of course
it was.)
(p.p.s. Thanks for Datasette!)",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/639/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
612382643,MDU6SXNzdWU2MTIzODI2NDM=,758,Question: Access to immutable database-path,2181410,open,0,,,6,2020-05-05T07:01:18Z,2020-05-28T08:23:27Z,,NONE,,"Hi Simon
Is there anywhere in the app-context where one can access the hashed urlpath of the database? Currently it's included in the template-context (`databases[0][""path"")` when rendering urls of the database (eg. `/db-44b06v9/cases`...), but where can I find the hashed url when rendering the index-page? I'm trying to avoid redirects. Thanks!",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/758/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
664485022,MDU6SXNzdWU2NjQ0ODUwMjI=,46,Feature: pull request reviews and comments,1326704,open,0,,,6,2020-07-23T13:43:45Z,2022-12-20T14:40:15Z,,NONE,,"Hi there! I saw your [presentation at Boston Python](https://www.meetup.com/bostonpython/events/271887195). I'm already a light user of Datasette (thank you!), but wasn't aware of this project.
I've been working on a ""pull request dashboard"" to get a comprehensive view of the state of open PR's, esp. related to reviews (i.e., pending, approved, changes requested). Currently it's a CLI command, but I thought a Datasette UI might be fun.
I see that PR's are available from the `issues` command, but I don't see reviews anywhere. From the [API docs](https://docs.github.com/en/rest/reference/pulls#reviews), it looks like there are separate endpoints for those (as well as pull requests in general). What do you think about adding that? Would you accept a PR? Any sense of the level of effort?",207052882,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
678760988,MDU6SXNzdWU2Nzg3NjA5ODg=,932,End-user documentation,9599,open,0,,3268330,6,2020-08-13T22:04:39Z,2022-03-08T15:20:48Z,,OWNER,,"Datasette's documentation is aimed at people who install and configure it.
What about end users of preconfigured and deployed Datasette instances?
Something that can be linked to from the Datasette UI would be really useful.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/932/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
718938889,MDU6SXNzdWU3MTg5Mzg4ODk=,5,Figure out how to display images from tags inline in Datasette,9599,open,0,,,6,2020-10-11T22:17:03Z,2020-10-16T20:16:28Z,,MEMBER,,"Relates to #1. Evernote XML looks like this:
```xml
This note includes two images.
The Python logo
The Evernote logo
```
That hash is the md5 we use to store resources. It should be possible to turn these into embedded image tags, especially if done in conjunction with the https://github.com/simonw/datasette-media plugin.",303218369,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
721068929,MDU6SXNzdWU3MjEwNjg5Mjk=,1020,Method for datasette.client() to forward on authentication,9599,open,0,,,6,2020-10-14T01:47:49Z,2020-10-19T22:45:01Z,,OWNER,,"I stumbled into this while working on Dogsheep Beta: the requests it re-dispatched through `TableView` did not carry authentication cookies, and since this was against a private instance they were thus denied.
https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L223-L231
This made me think that `datasette.client.get()` (which Dogsheep Beta will start using shortly) could benefit from some kind of utility mechanism for passing through the cookies and general authenticated state from the current request.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1020/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
743371103,MDU6SXNzdWU3NDMzNzExMDM=,1099,Support linking to compound foreign keys,9599,open,0,,,6,2020-11-15T23:23:17Z,2023-01-25T00:58:26Z,,OWNER,,Reported as a bug in #1098 because they caused 500 errors - but it would be even better if Datasette could hyperlink to related rows via compound foreign keys.,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1099/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
764059235,MDU6SXNzdWU3NjQwNTkyMzU=,1143,"More flexible CORS support in core, to encourage good security practices",114388,open,0,,3268330,6,2020-12-12T17:06:35Z,2022-02-13T17:41:17Z,,NONE,,"It would be nice if the `--cors` option accepted an origin regex to more securely allow secure local development.
As an example, Observable notebooks namespace every user's notebooks by their username and user content is served from username.observableusercontent.com, so you would set `--cors-origin username.observableusercontent.com` to restrict access to a local development Datasette instance to only your own notebooks, rather than exposing the data to any website that makes a request.
Thank you for all of your work on Datasette!",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1143/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
814595021,MDU6SXNzdWU4MTQ1OTUwMjE=,1241,Share button for copying current URL,7107523,open,0,,,6,2021-02-23T15:55:40Z,2023-08-24T20:09:52Z,,NONE,,"I use datasette in an `iframe` inside another HTML file that contains other ways to represent my data (mostly leaflets maps built with R on summarized data), and the datasette `iframe` is a tab in that page.
This particular use prevents users to access the full URLs of their datasette views and queries, which is a shame because the way datasette handles URLs to make every view or query easy to share is awesome. I know how to get the URL from the context menu of my browser, but I don't think many visitors would do it or even notice that datasette uses permalinks for pretty much every action they do. Would it be possible to add a ""Share link"" button to the interface, either in datasette itself or in a plugin?",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1241/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
991191951,MDU6SXNzdWU5OTExOTE5NTE=,1464,clean checkout & clean environment has test failures,51016,open,0,,,6,2021-09-08T14:16:23Z,2021-09-13T22:17:17Z,,CONTRIBUTOR,,"I followed the instructions [here](https://docs.datasette.io/en/stable/contributing.html#setting-up-a-development-environment), and even after running `python update-docs-help.py` I get the following failed tests -- any thoughts?
```
FAILED tests/test_api.py::test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3]
FAILED tests/test_api.py::test_searchmode[table_metadata1-_search=te*+AND+do*-expected_rows1]
FAILED tests/test_api.py::test_searchmode[table_metadata2-_search=te*+AND+do*&_searchmode=raw-expected_rows2]
```
This is with python 3.9.7 and lots of other packages, as in attached environment listing from `conda list`.
[conda-installed.txt](https://github.com/simonw/datasette/files/7129487/conda-installed.txt)
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1464/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1068791148,I_kwDOBm6k_c4_tHVs,1540,Idea: hover to reveal details of linked row,9599,open,0,,,6,2021-12-01T19:28:07Z,2021-12-09T23:38:39Z,,OWNER,,"![]()
Hovering over that could work a little bit like GitHub issue links:
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1540/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1121121305,I_kwDOBm6k_c5C0vQZ,1618,"Reconsider policy on blocking queries containing the string ""pragma""",770231,open,0,,,6,2022-02-01T19:39:46Z,2022-02-02T19:42:03Z,,NONE,,"First of all, thanks for creating this cool project, and also supporting publishing to various hosting services out of the box.
While testing out, I noticed legitimate queries such as
```
select * from books where title like 'Pragmatic%'
```
or
```
select * from books where title = 'The Pragmatic Programmer'
```
are blocked, due to the regular expression check here:
https://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L185
Example as seen from a Datasette instance:
https://fivethirtyeight.datasettes.com/polls?sql=select+*+from+books+where+title+like+%27Pragmatic%25%27%0D%0A
I'd propose a regular expression like
```
re.compile(f""pragma_(?!({'|'.join(allowed_pragmas)}))""),
```
instead of
```
re.compile(f""pragma(?!_({'|'.join(allowed_pragmas)}))""),
```
I can create a pull request with this change, unless the maintainers think it would allow unwanted queries to be executed.
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1618/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1193090967,I_kwDOBm6k_c5HHR-X,1699,Proposal: datasette query,25778,open,0,,,6,2022-04-05T12:36:43Z,2022-04-11T01:32:12Z,,CONTRIBUTOR,,"I started sketching out a plugin to add a `datasette query` subcommand to export data from the command line. This is based on discussions in #1356 and #1605. Before I get too far down this rabbit hole, I figure it's worth getting some feedback here (unless this should happen in `Discussions`). Here's what I'm thinking:
At its most basic, it will write the results of a query to STDOUT.
```sh
datasette query -d data.db 'select * from data' > results.json
```
This isn't much improvement over using [sqlite-utils](https://github.com/simonw/sqlite-utils). To make better use of datasette and its ecosystem, run `datasette query` using a canned query defined in a `metadata.yml` file.
For example, using the metadata file from [alltheplaces-datasette](https://github.com/eyeseast/alltheplaces-datasette/blob/main/metadata.yml):
```sh
cd alltheplaces-datasette
datasette query -d alltheplaces.db -m metadata.yml count_by_spider
```
That query would be good to get as CSV, and we can auto-discover metadata and databases in the current directory:
```sh
cd alltheplaces-datasette
datasette query count_by_spider -f csv
```
In this case, `count_by_spider` is a canned query defined on the `alltheplaces` database. If the same query is defined on multiple databases or its otherwise unclear which database `query` should use, pass the `-d` or `--database` option.
If a query takes parameters, I can pass them in at runtime, using the `--param` or `-p` option:
```sh
datasette query -d data.db -p value something 'select * from neighborhoods where some_column = :value'
```
I'm very interested in feedback on this, including whether it should be a plugin or in Datasette core. (I don't have a strong opinion about this, but I'm prototyping it as a plugin to start.)",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1699/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1439009231,I_kwDOBm6k_c5VxYnP,1884,Exclude virtual tables from datasette inspect,25778,open,0,,,6,2022-11-07T21:26:01Z,2022-11-21T04:40:56Z,,CONTRIBUTOR,,"Ran `inspect` on a spatialite database and got these warnings:
```
ERROR: conn=, sql = 'select count(*) from [SpatialIndex]', params = None: no such module: VirtualSpatialIndex
ERROR: conn=, sql = 'select count(*) from [ElementaryGeometries]', params = None: no such module: VirtualElementary
ERROR: conn=, sql = 'select count(*) from [KNN]', params = None: no such module: VirtualKNN
```
It still worked, but probably want to catch this.
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1884/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1661860507,PR_kwDOBm6k_c5N_bMw,2056,GitHub Action to lint Python code with ruff,3709715,open,0,,,6,2023-04-11T06:41:27Z,2023-04-15T14:24:46Z,,FIRST_TIME_CONTRIBUTOR,simonw/datasette/pulls/2056,"[Ruff](https://beta.ruff.rs/) supports [over 500 lint rules](https://beta.ruff.rs/docs/rules) and can be used to replace [Flake8](https://pypi.org/project/flake8/) (plus dozens of plugins), [isort](https://pypi.org/project/isort/), [pydocstyle](https://pypi.org/project/pydocstyle/), [yesqa](https://github.com/asottile/yesqa), [eradicate](https://pypi.org/project/eradicate/), [pyupgrade](https://pypi.org/project/pyupgrade/), and [autoflake](https://pypi.org/project/autoflake/), all while executing (in Rust) tens or hundreds of times faster than any individual tool.
The ruff Action uses minimal steps to run in ~5 seconds, rapidly providing intuitive GitHub Annotations to contributors.
----
:books: Documentation preview :books:: https://datasette--2056.org.readthedocs.build/en/2056/
",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2056/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1858228057,I_kwDOBm6k_c5uwk9Z,2147,Plugin hook for database queries that are run,18899,open,0,,,6,2023-08-20T18:43:50Z,2023-08-24T03:54:35Z,,NONE,,"I'm interested in making a plugin that saves every query that gets run to a table in the database. (I know about datasette-query-history but thought it would be good to have a server-side option.)
As far as I can tell reading the docs, there isn't really a hook setup to allow this.
Maybe I could hack it with some of the hooks that are passed requests, but that doesn't seem good.
I'm a little surprised this isn't possible, so I thought I would open an issue and see if that's a deeply considered decision or just ""haven't needed it yet."" I'm potentially interested in implementing the hook if the latter.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2147/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1876353656,I_kwDOBm6k_c5v1uJ4,2168,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(),9599,open,0,,,6,2023-08-31T21:42:04Z,2023-09-10T17:54:08Z,,OWNER,,"There's a long justification for why this might be needed here:
- https://github.com/simonw/datasette-auth-tokens/issues/10#issuecomment-1701820001
Short version: it would be neat if it was possible to stash some data on the `request` object such that a later plugin/middleware-type-thing could use that to influence the final returned response - similar to the kinds of things you can do with Django middleware.
The `asgi_wrapper()` mechanism doesn't have access to the request or response objects - it gets `scope` and can mess around with `receive` and `send`, but those are pretty low-level primitives.
Since Datasette has well-defined `request` and `response` objects now it might be nice to have a middleware layer that can manipulate those directly.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2168/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
447408527,MDU6SXNzdWU0NDc0MDg1Mjc=,483,Option to facet by date using month or year,9599,open,0,,,5,2019-05-23T01:25:29Z,2019-05-29T21:38:27Z,,OWNER,,"Facet by date (from #481) can take datetimes and facet them by the day component.
https://latest.datasette.io/fixtures/facetable?_facet_date=created
I'd like to also be able to facet by month or year.
I'm not sure what the best way to achieve this is. Could be two more Facet classes (YearFacet and MonthFacet) but I think it might be nicer if the existing DateFacet could take an optional argument that changed its behaviour. But... if I do that, do I expose it in the UI somewhere or is it only available to URL-hackers?",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/483/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
608512747,MDU6SXNzdWU2MDg1MTI3NDc=,14,Annotate photos using the Google Cloud Vision API,9599,open,0,,,5,2020-04-28T18:09:03Z,2020-04-28T18:19:06Z,,MEMBER,,"It can detect faces, run OCR, do image labeling (it knows what a lemur is!) and do object localization where it identifies objects and returns bounding polygons for them.",256834907,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14/reactions"", ""total_count"": 3, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,
670209331,MDU6SXNzdWU2NzAyMDkzMzE=,913,Mechanism for passing additional options to `datasette my.db` that affect plugins,9599,open,0,,,5,2020-07-31T20:38:26Z,2021-01-04T20:04:11Z,,OWNER,,"> It's a shame there's no obvious mechanism for passing additional options to `datasette my.db` that affect how plugins work.
>
>The only way I can think of at the moment is via environment variables:
>
> DATASETTE_INSERT_UNSAFE=1 datasette my.db
>
>This will have to do for the moment - it's ugly enough that people will at least know they are doing something unsafe, which is the goal here.
_Originally posted by @simonw in https://github.com/simonw/datasette-insert/issues/15#issuecomment-667346438_",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/913/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
675753042,MDU6SXNzdWU2NzU3NTMwNDI=,131,sqlite-utils insert: options for column types,9599,open,0,,,5,2020-08-09T18:59:11Z,2022-03-15T13:21:42Z,,OWNER,,"The `insert` command currently results in string types for every column - at least when used against CSV or TSV inputs.
It would be useful if you could do the following:
- automatically detects the column types based on eg the first 1000 records
- explicitly state the rule for specific columns
`--detect-types` could work for the former - or it could do that by default and allow opt-out using `--no-detect-types`
For specific columns maybe this:
sqlite-utils insert db.db images images.tsv \
--tsv \
-c id int \
-c score float",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/131/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
718540751,MDU6SXNzdWU3MTg1NDA3NTE=,1012,For 1.0 update trove classifier in setup.py,9599,open,0,,3268330,5,2020-10-10T05:52:08Z,2021-11-16T13:18:36Z,,OWNER,, Development Status :: 5 - Production/Stable,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1012/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
724759588,MDU6SXNzdWU3MjQ3NTk1ODg=,29,Add search highlighting snippets,9599,open,0,,,5,2020-10-19T16:00:48Z,2021-08-26T20:23:11Z,,MEMBER,,Like on https://til.simonwillison.net/til/search?q=Snippet,197431109,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,
732674148,MDU6SXNzdWU3MzI2NzQxNDg=,1062,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,9599,open,0,,3268330,5,2020-10-29T21:25:02Z,2022-09-28T14:09:54Z,,OWNER,,This can drive the upgrade of the `register_output_renderer` hook to be able to handle streaming all rows in a large query.,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1062/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
771608692,MDU6SXNzdWU3NzE2MDg2OTI=,14,UNIQUE constraint failed: workouts.id,1234956,open,0,,,5,2020-12-20T15:11:20Z,2023-07-10T14:46:52Z,,NONE,,"I'm getting an error on my initial attempt to import data:
```console
$ healthkit-to-sqlite 20201119\ healthkit\ export.zip healthkit.db
Importing from HealthKit [###################################-] 98% 00:00:01
Traceback (most recent call last):
File ""venv/bin/healthkit-to-sqlite"", line 8, in
sys.exit(cli())
File ""venv/lib/python3.9/site-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""venv/lib/python3.9/site-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""venv/lib/python3.9/site-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""venv/lib/python3.9/site-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""venv/lib/python3.9/site-packages/healthkit_to_sqlite/cli.py"", line 57, in cli
convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)
File ""venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py"", line 34, in convert_xml_to_sqlite
workout_to_db(el, db, zipfile)
File ""venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py"", line 57, in workout_to_db
pk = db[""workouts""].insert(record, alter=True, hash_id=""id"").last_pk
File ""venv/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1660, in insert
return self.insert_all(
File ""venv/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1778, in insert_all
self.insert_chunk(
File ""venv/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1588, in insert_chunk
result = self.db.execute(query, params)
File ""venv/lib/python3.9/site-packages/sqlite_utils/db.py"", line 213, in execute
return self.conn.execute(sql, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: workouts.id
```",197882382,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
778380836,MDU6SXNzdWU3NzgzODA4MzY=,4,Feature Request: Gmail,203343,open,0,,,5,2021-01-04T21:31:09Z,2021-03-04T20:54:44Z,,NONE,,"From takeout, I only exported my Gmail account. Ideally I could parse this into sqlite via this tool.",206649770,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
779088071,MDU6SXNzdWU3NzkwODgwNzE=,54,Archive import appears to be broken on recent exports,21148,open,0,,,5,2021-01-05T14:18:01Z,2023-01-04T11:06:55Z,,CONTRIBUTOR,,"I requested a Twitter export yesterday, and unfortunately they seem to have changed it such that `twitter-to-sqlite import` can't handle it anymore 😢
So far I've ran into two issues. The first was easy to work around, but the second will take more investigation. If I can find the time I'll keep working on it and update this issue accordingly.
The issues (so far):
### 1. Data seems to have moved to a `data/` subdirectory
Running `twitter-to-sqlite import` on the raw zip file reports a bunch of ""not yet implemented"" errors, and then exits without actually importing anything:
```
❯ twitter-to-sqlite import tarchive.db twitter.zip
...
data/manifest: not yet implemented
data/account-creation-ip: not yet implemented
data/account-suspension: not yet implemented
... (dozens of more lines like this, including critical stuff like data/tweets) ...
```
(`tarchive.db` now exists, but is empty)
Workaround: unpack the zip file, and run `twitter-to-sqlite import tarchive.db path/to/archive/data`
That gets further, but:
### 2. Some schema(s?) have changed
At least, the `blocks` schema seems different now:
```
❯ twitter-to-sqlite import tarchive.db archive/data
direct-messages-group: not yet implemented
branch-links: not yet implemented
periscope-expired-broadcasts: not yet implemented
direct-messages: not yet implemented
mute: not yet implemented
Traceback (most recent call last):
File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/bin/twitter-to-sqlite"", line 8, in
sys.exit(cli())
File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/cli.py"", line 772, in import_
archive.import_from_file(db, filepath.name, open(filepath, ""rb"").read())
File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py"", line 215, in import_from_file
to_insert = transformer(data)
File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py"", line 115, in lists_member
return {""lists-member"": _list_from_common(data)}
File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py"", line 200, in _list_from_common
for url in block[""userListInfo""][""urls""]:
KeyError: 'urls'
```
That's as far as I got before I needed to work on something else. I'll report back if I get further!",206156866,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
828858421,MDU6SXNzdWU4Mjg4NTg0MjE=,1258,Allow canned query params to specify default values,1385831,open,0,,,5,2021-03-11T07:19:02Z,2023-02-20T23:39:58Z,,NONE,,"If I call a canned query that includes named parameters, without passing any parameters, datasette runs the query anyway, resulting in an HTTP status code 400, and a visible error in the browser, with only a link back to home. This means that one of the default links on https://site/database/ will lead to a broken page with no apparent way out.
Is there any way to skip performing the query when parameters aren't supplied, but otherwise render the usual canned query page? Alternatively, can I supply default values for my parameters, either when defining my canned queries or when linking to the canned query page from the default database template.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1258/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
830567275,MDU6SXNzdWU4MzA1NjcyNzU=,1259,Research using CTEs for faster facet counts,9599,open,0,,,5,2021-03-12T22:19:49Z,2021-03-21T22:55:31Z,,OWNER,,"https://www.sqlite.org/changes.html#version_3_35_0
> Add support for the [MATERIALIZED](https://www.sqlite.org/lang_with.html#mathint) and [NOT MATERIALIZED](https://www.sqlite.org/lang_with.html#mathint) hints when specifying [common table expressions](https://www.sqlite.org/lang_with.html). The default behavior was formerly NOT MATERIALIZED, but is now changed to MATERIALIZED for CTEs that are used more than once.
If a CTE creates a table that is used multiple time in that query, SQLite will now default to creating a materialized table for the duration of that query.
This could be a big performance boost when applying faceting multiple times against the same query. Consider this example query:
```sql
WITH data as (
select
*
from
[global-power-plants]
),
country_long as (select
'country_long' as col, country_long as value, count(*) as c from data group by country_long
order by c desc limit 10
),
primary_fuel as (
select
'primary_fuel' as col, primary_fuel as value, count(*) as c from data group by primary_fuel
order by c desc limit 10
)
select * from primary_fuel union select * from country_long order by col, c desc
```
https://global-power-plants.datasettes.com/global-power-plants?sql=WITH+data+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Acountry_long+as+%28select+%0D%0A++%27country_long%27+as+col%2C+country_long+as+value%2C+count%28*%29+as+c+from+data+group+by+country_long%0D%0A++order+by+c+desc+limit+10%0D%0A%29%2C%0D%0Aprimary_fuel+as+%28%0D%0Aselect%0D%0A++%27primary_fuel%27+as+col%2C+primary_fuel+as+value%2C+count%28*%29+as+c+from+data+group+by+primary_fuel%0D%0A++order+by+c+desc+limit+10%0D%0A%29%0D%0Aselect+*+from+primary_fuel+union+select+*+from+country_long+order+by+col%2C+c+desc
Outputs:
col | value | c
-- | -- | --
country_long | United States of America | 8688
country_long | China | 4235
country_long | United Kingdom | 2603
country_long | Brazil | 2360
country_long | France | 2155
country_long | India | 1590
country_long | Germany | 1309
country_long | Canada | 1159
country_long | Spain | 829
country_long | Russia | 545
primary_fuel | Solar | 9662
primary_fuel | Hydro | 7155
primary_fuel | Wind | 5188
primary_fuel | Gas | 3922
primary_fuel | Coal | 2390
primary_fuel | Oil | 2290
primary_fuel | Biomass | 1396
primary_fuel | Waste | 1087
primary_fuel | Nuclear | 198
primary_fuel | Geothermal | 189
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1259/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
849220154,MDU6SXNzdWU4NDkyMjAxNTQ=,1286,Better default display of arrays of items,192568,open,0,,,5,2021-04-02T13:31:40Z,2021-06-12T12:36:15Z,,CONTRIBUTOR,,"Would be great to have template filters that convert array fields to bullets and/or delimited lists upon table display:
```
|to_bullets
|to_comma_delimited
|to_semicolon_delimited
```
or maybe:
```
|join_array(""bullet"")
|join_array(""bullet"",""square"")
|join_array("";"")
|join_array("","")
```
Keeping in mind that bullets show up in html as \ while other delimiting characters appear after the value.
Of course, the fields themselves would remain as facetable arrays. ",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1286/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
849396758,MDU6SXNzdWU4NDkzOTY3NTg=,1287,Upgrade to Python 3.9.4,9599,open,0,,,5,2021-04-02T18:43:15Z,2021-04-03T22:38:39Z,,OWNER,,Has some security fixes https://pythoninsider.blogspot.com/2021/04/python-393-and-389-are-now-available.html,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1287/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
920636216,MDU6SXNzdWU5MjA2MzYyMTY=,64,"feature: support ""events""",231498,open,0,,,5,2021-06-14T17:42:49Z,2021-06-15T00:48:37Z,,NONE,,"the GitHub API provides the ability to fetch all events for a given user, organization, or repository: https://docs.github.com/en/rest/reference/activity#list-events-for-the-authenticated-user
this would allow users to export all of the issue comments, new issues, etc. that they created. something which is currently missing from the GitHub takeout exports.",207052882,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
924748955,MDU6SXNzdWU5MjQ3NDg5NTU=,1380,Serve all db files in a folder,193463,open,0,,,5,2021-06-18T10:03:32Z,2021-11-13T08:09:11Z,,NONE,,"I tried to get the `serve` command to serve all the .db files in the `/mnt` folder but is seems that the server does not refresh the list of files.
In more detail:
* Starting datasette as a docker container with:
```
docker run -p 8001:8001 -v `pwd`:/mnt \
datasetteproject/datasette \
datasette -p 8001 -h 0.0.0.0 /mnt
```
* Datasette correctly serves all the *.db files found in the /mnt folder
* When the server is running, if I copy a new file in the $PWD folder, Datasette does not seem to see the new files, forcing me to restart Docker.
Is there an option/setting that I overlooked, or is this something missing?
BTW, the `--reload` setting, although at first glance is what you think you need, does not seem to do anything in regards of seeing all *.db files.
Thanks!",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1380/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
977323133,MDU6SXNzdWU5NzczMjMxMzM=,1445,Ability to search for text across all columns in a table,9599,open,0,,,5,2021-08-23T18:50:48Z,2021-08-23T19:10:17Z,,OWNER,,"When I'm working with new data I often find myself wanting to run a search for text embedded in ANY of the columns of a table, without having to even fully understand the schema first.
I figured out a trick for doing that using a SQL-generated SQL query here: https://til.simonwillison.net/datasette/search-all-columns-trick
But maybe this should be a core Datasette feature? Or a plugin?",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1445/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1033678984,PR_kwDOBm6k_c4tjgJ8,1495,Allow routes to have extra options,536941,open,0,,,5,2021-10-22T15:00:45Z,2021-11-19T15:36:27Z,,CONTRIBUTOR,simonw/datasette/pulls/1495,"Right now, datasette routes can only be a 2-tuple of `(regex, view_fn)`.
If it was possible for datasette to handle extra options, like [standard Django does](https://docs.djangoproject.com/en/3.2/topics/http/urls/#passing-extra-options-to-view-functions), it would add flexibility for plugin authors.
For example, if extra options were enabled, then it would be easy to make a single table the home page (#1284). This plugin would accomplish it.
```python
from datasette import hookimpl
from datasette.views.table import TableView
@hookimpl
def register_routes(datasette):
return [
(r""^/$"", TableView.as_view(datasette), {'db_name': 'DB_NAME',
'table': 'TABLE_NAME'})
]
```
",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1495/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1066563554,I_kwDOCGYnMM4_knfi,346,Way to test SQLite 3.37 (and potentially other versions) in CI,9599,open,0,,,5,2021-11-29T22:21:06Z,2021-11-29T23:12:49Z,,OWNER,,"> Need to figure out a good pattern for testing this in CI too - it will currently skip the new tests if it doesn't have SQLite 3.37 or higher.
_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982076924_",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/346/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1077620955,I_kwDOBm6k_c5AOzDb,1549,Redesign CSV export to improve usability,536941,open,0,,3268330,5,2021-12-11T19:02:12Z,2022-04-04T11:17:13Z,,CONTRIBUTOR,,"*Original title: Set content type for CSV so that browsers will attempt to download instead opening in the browser*
Right now, if the user clicks on the CSV related to a table or a query, the response header for the content type is
""content-type: text/plain; charset=utf-8""
Most browsers will try to open a file with this content-type in the browser.
This is not what most people want to do, and lots of folks don't know that if they want to download the CSV and open it in the a spreadsheet program they next need to save the page through their browser.
It would be great if the response header could be something like
```
'Content-type: text/csv');
'Content-disposition: attachment;filename=MyVerySpecial.csv');
```
which would lead browsers to open a download dialog.
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1549/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1114628238,I_kwDOBm6k_c5Cb-CO,1613,Improvements to help make Datasette a better tool for learning SQL,9599,open,0,,,5,2022-01-26T04:56:07Z,2022-01-26T16:41:46Z,,OWNER,,Tracking issue for the general goal of making Datasette a better tool for learning SQL.,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1613/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1121583414,I_kwDOBm6k_c5C2gE2,1619,JSON link on row page is 404 if base_url setting is used,9599,open,0,,,5,2022-02-02T07:09:53Z,2023-03-24T15:38:04Z,,OWNER,,"On my local environment:
datasette fixtures.db -p 3344 --setting base_url /foo/bar/
Then hit http://127.0.0.1:3344/foo/bar/fixtures/table%2Fwith%2Fslashes.csv/3
![]()
But... that `json` link goes here, which is a 404:
http://127.0.0.1:3344/foo/bar/foo/bar/fixtures/table%2Fwith%2Fslashes.csv/3?_format=json",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1619/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1128466114,I_kwDOCGYnMM5DQwbC,406,Creating tables with custom datatypes,82988,open,0,,,5,2022-02-09T12:16:31Z,2022-09-15T18:13:50Z,,NONE,,"Via https://stackoverflow.com/a/18622264/454773 I note the ability to register custom handlers for novel datatypes that can map into and out of things like sqlite `BLOB`s.
From a quick look and a quick play, I didn't spot a way to do this in `sqlite_utils`?
For example:
```python
# Via https://stackoverflow.com/a/18622264/454773
import sqlite3
import numpy as np
import io
def adapt_array(arr):
""""""
http://stackoverflow.com/a/31312102/190597 (SoulNibbler)
""""""
out = io.BytesIO()
np.save(out, arr)
out.seek(0)
return sqlite3.Binary(out.read())
def convert_array(text):
out = io.BytesIO(text)
out.seek(0)
return np.load(out)
# Converts np.array to TEXT when inserting
sqlite3.register_adapter(np.ndarray, adapt_array)
# Converts TEXT to np.array when selecting
sqlite3.register_converter(""array"", convert_array)
```
```python
from sqlite_utils import Database
db = Database('test.db')
# Reset the database connection to used the parsed datatype
# sqlite_utils doesn't seem to support eg:
# Database('test.db', detect_types=sqlite3.PARSE_DECLTYPES)
db.conn = sqlite3.connect(db_name, detect_types=sqlite3.PARSE_DECLTYPES)
# Create a table the old fashioned way
# but using the new custom data type
vector_table_create = """"""
CREATE TABLE dummy
(title TEXT, vector array );
""""""
cur = db.conn.cursor()
cur.execute(vector_table_create)
# sqlite_utils doesn't appear to support custom types (yet?!)
# The following errors on the ""array"" datatype
""""""
db[""dummy""].create({
""title"": str,
""vector"": ""array"",
})
""""""
```
We can then add / retrieve records from the database where the datatype of the `vector` field is a custom registered `array` type (which is to say, a `numpy` array):
```python
import numpy as np
db[""dummy""].insert({'title':""test1"", 'vector':np.array([1,2,3])})
for row in db.query(""SELECT * FROM dummy""):
print(row['title'], row['vector'], type(row['vector']))
""""""
test1 [1 2 3]
""""""
```
It would be handy to be able to do this idiomatically in `sqlite_utils`.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/406/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1203943272,I_kwDOBm6k_c5Hwrdo,1713,Datasette feature for publishing snapshots of query results,9599,open,0,,,5,2022-04-14T01:42:00Z,2022-07-04T05:16:35Z,,OWNER,,"https://twitter.com/simonw/status/1514392335718645760
> Maybe [@datasetteproj](https://twitter.com/datasetteproj) should grow a feature that lets you cache the results of a query and give that snapshot a stable permalink
>
> A plugin that publishes the JSON output of a query to an S3 bucket would be pretty neat... especially if it could also be configured to re-publish the results on a schedule
A lot of people said they would find this useful.
Probably going to build this as a plugin.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1713/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1236693079,I_kwDOCGYnMM5JtnBX,432,"Support `rows_where()`, `delete_where()` etc for attached alias databases",11597658,open,0,,,5,2022-05-16T06:38:58Z,2022-06-14T22:16:48Z,,NONE,,"Hi,
I noticed `rows_where()` doesn't return any rows from tables which are from attached databases. The `exists()` function returns false. As far as I can see this is because the `table_names()` function only looks for table names in the current database and not in attached (or temp) databases.
Besides, `rows_where()`, also `insert_all()` and `delete_where()` didn't do what I was expecting because of this. For the moment I've patched `table_names()` for myself, see below but I'm not sure what the total impact is on the other functions like lookup truncate etc which all use `exists()`. Also `view_names()` doesn't look for views in attached or temp databases.
```python
def table_names(self, fts4: bool = False, fts5: bool = False) -> List[str]:
""A list of string table names in this database.""
where = [""type = 'table'""]
if fts4:
where.append(""sql like '%USING FTS4%'"")
if fts5:
where.append(""sql like '%USING FTS5%'"")
dbs = [x[1] for x in self.execute('pragma database_list').fetchall()]
lst=[]
for db in dbs:
sql = ""select name from {} where {}"".format(db+"".sqlite_master"","" AND "".join(where))
lst.extend(r[0] for r in self.execute(sql).fetchall())
return lst
```",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/432/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1271426387,I_kwDOCGYnMM5LyG1T,444,CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool,9599,open,0,,,5,2022-06-14T22:22:47Z,2022-07-07T16:39:18Z,,OWNER,,"> I forgot to add equivalents of `extras_key=` and `ignore_extras=` to the CLI tool - will do that in a separate issue.
_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155767915_",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/444/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1310243385,I_kwDOCGYnMM5OGLo5,456,feature request: pivot command,536941,open,0,,,5,2022-07-20T00:58:08Z,2022-07-20T17:50:50Z,,CONTRIBUTOR,,pivoting long-format table to wide-format tables is pretty common and kind of pain. would love to see this feature in sqlite-utils!,140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/456/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1373224657,I_kwDOCGYnMM5R2b7R,488,`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float,9599,open,0,,,5,2022-09-14T15:51:30Z,2022-12-23T17:38:55Z,,OWNER,,"```
/tmp % echo ""id,age,weight\n1,3,2.5\n2,,"" | sqlite-utils insert test.db test - --csv
/tmp % sqlite-utils schema test.db
CREATE TABLE [test] (
[id] TEXT,
[age] TEXT,
[weight] TEXT
);
/tmp % sqlite-utils transform test.db test --type age integer --type weight float
/tmp % sqlite-utils schema test.db
CREATE TABLE ""test"" (
[id] TEXT,
[age] INTEGER,
[weight] FLOAT
);
/tmp % sqlite-utils rows test.db test
[{""id"": ""1"", ""age"": 3, ""weight"": 2.5},
{""id"": ""2"", ""age"": """", ""weight"": """"}]
```
It would be neat if this resulted in the following instead:
```
{""id"": ""2"", ""age"": null, ""weight"": null}
```
Related Discord discussion: https://discord.com/channels/823971286308356157/823971286941302908/1019635490833567794",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/488/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1432037325,I_kwDOBm6k_c5VWyfN,1879,Make it easier to fix URL proxy problems,9599,open,0,,,5,2022-11-01T20:19:23Z,2022-11-01T20:33:52Z,,OWNER,,"This came up on Discord again today: figuring out how to run Datasette behind a proxy that might hide the incoming Host: header (and strip HTTPS) is really hard!
https://discord.com/channels/823971286308356157/823971286941302908/1037012475322847263",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1879/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1605481359,PR_kwDOBm6k_c5LDwrF,2031,Expand foreign key references in row view as well,82332573,open,0,,,5,2023-03-01T18:43:09Z,2023-03-24T18:35:25Z,,FIRST_TIME_CONTRIBUTOR,simonw/datasette/pulls/2031,"Unlike the table view, the single row view does not resolve foreign key references into labels. This patch extracts the foreign key reference expansion code from TableView.data() into a standalone function that is then called by both TableView.data() and RowView.data().
----
:books: Documentation preview :books:: https://datasette--2031.org.readthedocs.build/en/2031/
",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2031/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1818838294,I_kwDOCGYnMM5saUUW,578,Plugin hook for adding new output formats,9599,open,0,,,5,2023-07-24T17:29:18Z,2023-08-07T15:41:49Z,,OWNER,,"> What would it take to add a format hook? I'm still thinking about my GIS workflow, and being able to do `sqlite-utils query ... --geojson` would be nice. It's the one place my Datasette workflow is messy, having to do `datasette . --get /path/to/query.geojson --setting max_rows_returned 10000 --load-extension spatialite`.
> I know the current pattern is `--csv`, but maybe `--format geojson` is more future-proof.
https://discord.com/channels/823971286308356157/997738192360964156/1133076679011602432",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/578/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
309047460,MDU6SXNzdWUzMDkwNDc0NjA=,188,Ability to bundle metadata and templates inside the SQLite file,9599,open,0,,,4,2018-03-27T16:42:07Z,2020-12-04T17:18:34Z,,OWNER,,"One of the nicest qualities of SQLite as a data format is that you get a single file which you can then backup or share with other people.
Datasette breaks this a little once you start including custom metadata.json or template files and CSS.
It would be cool if there was an optional mechanism for baking that extra configuration into the SQLite file itself. That way entire datasette mini-applications (including canned queries and custom HTML and CSS) could be constructed as single .db files.
Since datasette configuration is all file-based, one way to achieve that would be to support a ""datasette_files"" table which, if present is used to search for file contents by path.
This is inline with the philosophy described by https://www.sqlite.org/appfileformat.html
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/188/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
335200136,MDU6SXNzdWUzMzUyMDAxMzY=,327,Explore if SquashFS can be used to shrink size of packaged Docker containers,9599,open,0,,,4,2018-06-24T18:15:16Z,2022-02-17T23:37:24Z,,OWNER,,"Inspired by this article: https://cldellow.com/2018/06/22/sqlite-parquet-vtable.html#sqlite-database-indexed--squashed
https://en.wikipedia.org/wiki/SquashFS is ""a compressed read-only file system for Linux"" - which means it could be a really nice fit for Datasette and its read-only SQLite databases.
It would be interesting to explore a Dockerfile recipe that used SquashFS to compress the SQLite database file that was bundled up by `datasette package` and friends.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/327/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
377155320,MDU6SXNzdWUzNzcxNTUzMjA=,370,Integration with JupyterLab,82988,open,0,,,4,2018-11-04T13:57:13Z,2022-09-29T08:17:47Z,,CONTRIBUTOR,,"I just watched a demo video for the [JupyterLab Chart Editor](https://www.crowdcast.io/e/introducing-JupyterLab-Chart-Editor/) which wraps the plotly chart editor app in a JupyterLab panel and lets you open a plotly chart JSON file in that editor. Essentially, it pops an HTML app into a panel in JupyterLab, and I think registers the app as a file viewer for a particular file type. (I'm not completely taken by it, tbh, because it means you can do irreproducible things to the chart definition file, but that's another issue).
JupyterLab extensions can also open files from a dialogue as the iframe/html previewer shows: https://github.com/timkpaine/jupyterlab_iframe.
This made me wonder about what `datasette` integration with JupyterLab might do.
For example, by right-clicking on a CSV file (for which there is already a CSV table view) in the file browser, offer a *View / Run as datasette* file viewer option that will:
- run the CSV file through `csvs-to-sqlite`;
- launch the `datasette` server and display the `datasette` view in a JupyterLab panel.
(? Create a new SQLite db for each CSV file and launch each datasette view on a new port? Or have a JupyterLab (session?) SQLite db that stores all `datasette` viewed CSVs and runs on a single port?)
As a freebie, the `datasette` API would allow you to run efficient SQL queries against the file eg using using `pandas.read_sql()` queries in a notebook in the same space.
Related:
- [JupyterLab extensions docs](https://jupyterlab.readthedocs.io/en/stable/user/extensions.html)
- a [cookiecutter for wrting JupyterLab extensions using Javascript](https://github.com/jupyterlab/extension-cookiecutter-js)
- a [cookiecutter for writing JupyterLab extensions using Typescript](https://github.com/jupyterlab/extension-cookiecutter-ts)
- tutorial: [Let’s Make an xkcd JupyterLab Extension](https://jupyterlab.readthedocs.io/en/stable/developer/xkcd_extension_tutorial.html)",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/370/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
443021509,MDU6SXNzdWU0NDMwMjE1MDk=,461,Paginate + search for databases/tables on the homepage,9599,open,0,,3268330,4,2019-05-11T18:05:34Z,2020-12-17T22:14:46Z,,OWNER,,Split out from #460 - in order to support large numbers of connected databases the homepage needs to be paginated.,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/461/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
455852801,MDU6SXNzdWU0NTU4NTI4MDE=,507,Every datasette plugin on the ecosystem page should have a screenshot,9599,open,0,,,4,2019-06-13T17:02:51Z,2020-09-17T02:47:35Z,,OWNER,,https://github.com/simonw/datasette/blob/master/docs/ecosystem.rst,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/507/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
481885279,MDU6SXNzdWU0ODE4ODUyNzk=,569,More advanced connection pooling,9599,open,0,,,4,2019-08-17T13:20:41Z,2019-10-02T22:44:37Z,,OWNER,,"We need a much smarter way of handling database connections.
Today, connections are simple: Datasette runs a number of threads (defaults to 3) and each thread gets a threadlocal read-only (or immutable) connection to each attached database - opened on demand.
For Datasette Library (#417) I want to support potentially hundreds of attached databases. Datasette Edit (#567) is going to introduce a need for writable connections too.
I'd also like to be able to run joins across multiple databases (#283) which further complicates things.
Supporting thousands of open SQLite connections at once feels like it won't provide good enough performance (though I should benchmark that to be sure). Some kind of connection pooling is likely to be necessary.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/569/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
548591089,MDU6SXNzdWU1NDg1OTEwODk=,657,Allow creation of virtual tables at startup,1055831,open,0,,,4,2020-01-12T16:10:55Z,2021-01-15T20:24:35Z,,NONE,,"Hi,
I've been experimenting with SQLite reading from huge datasets using this excellent Parquet extension from @cldellow.
https://cldellow.com/2018/06/22/sqlite-parquet-vtable.html
https://github.com/cldellow/sqlite-parquet-vtable
This works really well, but I was keen to see if I could combine datasette with this. Having previously experimented with the spatialite extension I knew that datasette supports loading extensions in the underlying sqlite instance. However I hit a blocker as the current design only allows SELECT statements to be executed and so I am unable to execute the crucial
CREATE VIRTUAL TABLE .........
command that is required to load the data from the parquet file into the table.
It seems like this would be a simple-ish change, but I don't know enough about the architecture of datasette to start implementing this myself? Could this be done as a datasette plugin? or would this require more fundamental changes at initialisation time?
My thoughts are that something at init time could detect that the user was loading a *.parquet file and then switch to a mode were it loads that via the ""CREATE VIRTUAL TABLE..."" rather than loading the *.db file in the default case??
I'm happy to contribute code and testing, I just need some pointers on the best approach.
Thanks
Darren",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/657/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
615626118,MDU6SXNzdWU2MTU2MjYxMTg=,22,Try out ExifReader,9599,open,0,,,4,2020-05-11T06:32:13Z,2020-05-14T05:59:53Z,,MEMBER,,"https://pypi.org/project/ExifReader/
New fork that should be able to handle EXIF in HEIC files.
Forked here: https://github.com/ianare/exif-py/issues/102#issuecomment-626376522
Refs #3 ",256834907,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
648749062,MDExOlB1bGxSZXF1ZXN0NDQyNTA1MDg4,883,Skip counting hidden tables,3243482,open,0,,,4,2020-07-01T07:38:08Z,2020-07-02T00:25:44Z,,CONTRIBUTOR,simonw/datasette/pulls/883,"Potential fix for https://github.com/simonw/datasette/issues/859.
Disabling table counts for hidden tables speeds up database page quite a bit. In my setup it reduced load time by 2/3 (~300 -> ~90ms)",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/883/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
703246031,MDU6SXNzdWU3MDMyNDYwMzE=,51,github-to-sqlite should handle rate limits better,9599,open,0,,,4,2020-09-17T04:01:50Z,2022-10-14T16:34:07Z,,MEMBER,,From #50 - right now it will crash with an error of it hits the rate limit. Since the rate limit information (including reset time) is available in the headers it could automatically sleep and try again instead.,207052882,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
706001517,MDU6SXNzdWU3MDYwMDE1MTc=,163,Idea: conversions= could take Python functions,9599,open,0,,,4,2020-09-22T00:37:12Z,2021-12-20T00:56:52Z,,OWNER,,"Right now you use `conversions=` like this:
```python
db[""example""].insert({
""name"": ""The Bigfoot Discovery Museum""
}, conversions={""name"": ""upper(?)""})
```
How about if you could optionally provide a Python function (or a lambda) like this?
```python
db[""example""].insert({
""name"": ""The Bigfoot Discovery Museum""
}, conversions={""name"": lambda s: s.upper()})
```
This would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/163/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
718238967,MDU6SXNzdWU3MTgyMzg5Njc=,1003,from_json jinja2 filter,649467,open,0,,,4,2020-10-09T15:30:58Z,2020-10-09T17:17:07Z,,NONE,,"When JSON fields are rendered in a jinja2 template, it is handy to be able to manipulate them as data (e.g., iterate over an array of values).
Ansible has a ""from_json"" function, which just called json.loads. It's a trivial as a datasette plugin, but it seems generally useful. Does it makes sense to add it directly into the app?",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1003/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
727848625,MDU6SXNzdWU3Mjc4NDg2MjU=,12,"Some workout columns should be float, not text",9599,open,0,,,4,2020-10-23T02:47:02Z,2022-06-23T04:35:02Z,,MEMBER,,"Columns `duration`, `totalDistance` and `totalEnergyBurned` should be converted to float.
https://github.com/dogsheep/healthkit-to-sqlite/blob/71e36e1cf034b96de2a8e6652265d782d3fdf63b/healthkit_to_sqlite/utils.py#L50-L57",197882382,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
756876238,MDExOlB1bGxSZXF1ZXN0NTMyMzQ4OTE5,1130,Fix footer not sticking to bottom in short pages,3243482,open,0,,,4,2020-12-04T07:29:01Z,2021-06-15T13:27:48Z,,CONTRIBUTOR,simonw/datasette/pulls/1130,Fixes https://github.com/simonw/datasette/issues/1129,107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1130/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
779156520,MDU6SXNzdWU3NzkxNTY1MjA=,1175,Use structlog for logging,9599,open,0,,,4,2021-01-05T15:11:36Z,2022-07-26T12:52:10Z,,OWNER,,To solve #241 JSON logging.,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1175/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
789336592,MDU6SXNzdWU3ODkzMzY1OTI=,1195,"view_name = ""query"" for the query page",9599,open,0,,,4,2021-01-19T20:21:36Z,2021-01-25T04:40:08Z,,OWNER,,It uses `view_name` of `database` at the moment which isn't as useful.,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1195/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
802513359,MDU6SXNzdWU4MDI1MTMzNTk=,1217,Possible to deploy as a python app (for Rstudio connect server)?,6165713,open,0,,,4,2021-02-05T22:21:24Z,2022-11-04T11:37:52Z,,NONE,,"Is it possible to deploy a `datasette` application as a python web app?
In my enterprise, I have option to deploy python apps via [Rstudio Connect](https://github.com/rstudio/rsconnect-python), and I would like to publish a `datasette` dashboard for sharing.
I welcome any pointers to converting `datasette serve` into a python app that can be run as something like `python datasette.py --my_data.db`",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1217/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
803338729,MDU6SXNzdWU4MDMzMzg3Mjk=,33,photo-to-sqlite: command not found,11855322,open,0,,,4,2021-02-08T08:42:57Z,2021-02-12T15:00:44Z,,NONE,,"Having installed in a venv I get:
```
(venv) (base) Robins-MacBook:datasette robin$ photo-to-sqlite apple-photos photos.db
-bash: photo-to-sqlite: command not found
```",256834907,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
838382890,MDU6SXNzdWU4MzgzODI4OTA=,1273,Refresh SpatiaLite documentation,9599,open,0,,,4,2021-03-23T06:05:55Z,2022-01-20T21:28:50Z,,OWNER,,https://docs.datasette.io/en/0.55/spatialite.html was written before I had tools like [geojson-to-sqlite](https://datasette.io/tools/geojson-to-sqlite) and [shapefile-to-sqlite](https://datasette.io/tools/shapefile-to-sqlite).,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1273/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
845794436,MDU6SXNzdWU4NDU3OTQ0MzY=,1284,Feature or Documentation Request: Individual table as home page template,192568,open,0,,,4,2021-03-31T03:56:17Z,2021-11-04T03:15:01Z,,CONTRIBUTOR,,"It would be great to have a sample showing how to move a single database that has a single table, to the index page. I'm trying it now, and find there is a real depth of Datasette and Python understanding that's required to be successful.
I've got all the basic jinja concepts down... variables, template control structures, template inheritance, template overrides, css, html, the --template-dir and --static arguments, etc.
But copying the table.html file to index.html doesn't work. There are undocumented functions and filters... I can figure some of them out (yay, url_builder.py and utils/__init__.py!) but it's a slog better handled by a much stronger Python developer.
One sample would make a world of difference. The ideal form of this documentation would be a diff between the default table.html and how that would look if essentially moved to index.html. The use case is for everyone who wants to create a public-facing website to explore a single table at the root directory. (Maybe a second bit of documentation for people who have a single database with multiple tables.)
(Hmm... might be cool to have a setting for that, where it happens automagically! If only one table, then home page is at the table level. if only one database, then home page is at the database level.... as an option.)
I suppose I could ignore this, and somehow do this in the DNS settings once I hook up Vercel to a domain name, maybe.. and remove the breadcrumbs in table.html... but for now, a documentation request in the form of a diff... for viewing a single table (or a single database) at the root.
(Actually, there's probably room for a whole expanded section on templates. Noticed some nice table metadata in one of the datasette examples, for instance... Hmm... maybe a whole library of solutions in one place... maybe a documentation hackathon! If that's of interest, of course it's a separate issue. )
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1284/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
855446829,MDExOlB1bGxSZXF1ZXN0NjEzMTc4OTY4,1296,Dockerfile: use Ubuntu 20.10 as base,82332573,open,0,,,4,2021-04-12T00:23:32Z,2021-07-20T08:52:13Z,,FIRST_TIME_CONTRIBUTOR,simonw/datasette/pulls/1296,"This PR changes the main Dockerfile to use ubuntu:20.10 as base image instead of python:3.9.2-slim-buster (itself based on debian:buster-slim).
The Dockerfile is essentially the one from https://github.com/simonw/datasette/issues/1249#issuecomment-803698983 with some additional cleanups to slim it down.
This fixes a couple of issues:
1. The SQLite version in Debian Buster (2.6.0) doesn't support generated columns
2. Installing SpatiaLite from the Debian sid repositories has the side effect of also installing updates to libc and libstdc++ from sid.
As a bonus, the Docker image becomes smaller:
```
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
datasette 0.56-ubuntu f7aca255140a 5 hours ago 212MB
datasetteproject/datasette 0.56 efb3b282f390 13 days ago 258MB
```
### Reproduction of the first issue
```
$ curl -O https://latest.datasette.io/fixtures.db
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 260k 0 260k 0 0 489k 0 --:--:-- --:--:-- --:--:-- 489k
$ docker run -v `pwd`:/mnt datasetteproject/datasette:0.56 datasette /mnt/fixtures.db
Traceback (most recent call last):
File ""/usr/local/bin/datasette"", line 8, in
sys.exit(cli())
File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""/usr/local/lib/python3.9/site-packages/datasette/cli.py"", line 544, in serve
asyncio.get_event_loop().run_until_complete(check_databases(ds))
File ""/usr/local/lib/python3.9/asyncio/base_events.py"", line 642, in run_until_complete
return future.result()
File ""/usr/local/lib/python3.9/site-packages/datasette/cli.py"", line 584, in check_databases
await database.execute_fn(check_connection)
File ""/usr/local/lib/python3.9/site-packages/datasette/database.py"", line 155, in execute_fn
return await asyncio.get_event_loop().run_in_executor(
File ""/usr/local/lib/python3.9/concurrent/futures/thread.py"", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File ""/usr/local/lib/python3.9/site-packages/datasette/database.py"", line 153, in in_thread
return fn(conn)
File ""/usr/local/lib/python3.9/site-packages/datasette/utils/__init__.py"", line 892, in check_connection
for r in conn.execute(
sqlite3.DatabaseError: malformed database schema (generated_columns) - near ""AS"": syntax error
```
Here is the SQLite version:
```
$ docker run -v `pwd`:/mnt -it datasetteproject/datasette:0.56 /bin/bash
root@d9220d3b95dd:/# python3
Python 3.9.2 (default, Mar 27 2021, 02:50:26)
[GCC 8.3.0] on linux
Type ""help"", ""copyright"", ""credits"" or ""license"" for more information.
>>> import sqlite3
>>> sqlite3.version
'2.6.0'
```
### Reproduction of the second issue
```
$ docker build . -t datasette --build-arg VERSION=0.55
[...snip...]
The following packages will be upgraded:
libc-bin libc6 libstdc++6
[...snip...]
Unpacking libc6:amd64 (2.31-11) over (2.28-10) ...
[...snip...]
Unpacking libstdc++6:amd64 (10.2.1-6) over (8.3.0-6) ...
[...snip...]
```
Both libc and libstdc++ are backwards compatible, so the image still works, but it will result in a combination of libraries and Python versions that exists only in the Datasette image, so it's likely untested. In addition, since Debian sid is an always-changing rolling-release, the versions of libc, libstdc++, Spatialite, and their dependencies change frequently, so the library versions in the Datasette image will depend on the day when it was built.
",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1296/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
855476501,MDU6SXNzdWU4NTU0NzY1MDE=,1298,improve table horizontal scroll experience,192568,open,0,,,4,2021-04-12T01:55:16Z,2022-08-30T21:11:49Z,,CONTRIBUTOR,,"Wide tables aren't a huge problem if you know to click and drag right. But it's not at all obvious to do that. (it also tends to blue-select any content as it's dragging.) Depending on column widths, public users might entirely miss all the columns to the right.
There is a scrollbar at the bottom of the table, but I'm displaying ALL my records because it's the only way for datasette-vega to make accurate charts. So that bottom scrollbar is likely to be missed. I wonder if some sort of javascript-y mouseover to an arrow might help, similar to those seen in image carousels. Ah: here's a perfect example:
1. Visit http://google.com
2. Search for: animals endangered
3. Note the 'g-right-button' (in the code) that looks like a right-facing caret in a circle.
4. Click on that and the carousel scrolls right (and 'g-left-button' appears on the left).
Might be tricky to do that on a table, rather than a one-row carousel, but it's worth experimenting with.
Another option is just to put the scrollbars at the top of the table, too.
Meantime, I'm trying to build a button like the ""View/hide all columns on https://salaries.news.baltimoresun.com/salaries-be494cf/2019+Maryland+state+salaries
Might be nice to have that available by default, with settings in the metadata showing which are on by default.
(I saw some other closed issues related to horizontal scrolling, and admit I don't entirely understand them. For instance, the animated gif at https://github.com/simonw/datasette/issues/998#issuecomment-714117534 confuses me. )
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1298/reactions"", ""total_count"": 4, ""+1"": 4, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
863884805,MDU6SXNzdWU4NjM4ODQ4MDU=,1304,"Document how to send multiple values for ""Named parameters"" ",9308268,open,0,,,4,2021-04-21T13:19:06Z,2021-12-08T03:23:14Z,,NONE,,"https://docs.datasette.io/en/stable/sql_queries.html#named-parameters
I thought that I had seen an example of how to do this example below, but I can't seem to find it
```sql
select
*
from
bib
where
bib.bib_record_num in (1008088,1008092)
```
```sql
select
*
from
bib
where
bib.bib_record_num in (:bib_record_numbers)
```
https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-204d100?sql=select%0D%0A++*%0D%0Afrom%0D%0A++bib%0D%0Awhere%0D%0A++bib.bib_record_num+in+%28%3Abib_record_numbers%29&bib_record_numbers=1008088%2C1008092
Or, maybe this isn't a fully supported feature.
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1304/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
915421499,MDU6SXNzdWU5MTU0MjE0OTk=,267,row.update() or row.pk,12721157,open,0,,,4,2021-06-08T19:56:00Z,2021-06-22T17:27:27Z,,NONE,,"Hi,
fantastic framework for working with Sqlite3 databases!!!
I tried to update spezific rows in a table and used
for row in db[tablename]:
newValue = row[""counter""] * row[""prize""]
row.update({""Fieldname"": newValue})
print(row)
This updates the value in the printet row, but not in the database. So I switched to
db[tablename].update(id, {""Filedname"": newValue})
This works fine. But row.update would be nicer, because no need for the id (its that row), no need for the tablename and the db (all defined in the for row ... loop).
Thx
",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/267/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
952179830,MDU6SXNzdWU5NTIxNzk4MzA=,2,Command for fetching Hacker News threads from the search API,9599,open,0,,,4,2021-07-25T02:00:45Z,2021-07-25T03:12:57Z,,MEMBER,,"I want to be able to fetch every item for a domain, e.g. https://news.ycombinator.com/from?site=simonwillison.net",248903544,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
959137143,MDU6SXNzdWU5NTkxMzcxNDM=,1415,feature request: document minimum permissions for service account for cloudrun,536941,open,0,,,4,2021-08-03T13:48:43Z,2023-11-05T16:46:59Z,,CONTRIBUTOR,,"Thanks again for such a powerful project.
For deploying to cloudrun from github actions, I'd like to create a service account with minimal permissions.
It would be great to document what those minimum permission that need to be set in the IAM.
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1415/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
963527045,MDU6SXNzdWU5NjM1MjcwNDU=,1424,Document exceptions that can be raised by db.execute() and friends,9599,open,0,,,4,2021-08-08T22:23:25Z,2021-08-08T22:27:31Z,,OWNER,,Not currently covered here: https://docs.datasette.io/en/stable/internals.html#await-db-execute-sql,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1424/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1065429936,I_kwDOBm6k_c4_gSuw,1532,Use datasette-table Web Component to guide the design of the JSON API for 1.0,9599,open,0,,3268330,4,2021-11-28T20:37:18Z,2022-03-16T20:13:34Z,,OWNER,,"I realized that one of the reasons I'm having trouble committing to nailing down the JSON API for 1.0 is that I don't use it much myself - I use the `?_shape=array` one quite often, but I don't have any projects that are using the default, more fully-featured API.
As an experiment I built a Web Component for embedding Datasette tables on pages - https://github.com/simonw/datasette-table - and I think it's actually going to be a really useful tool for helping me dog food the v1.0 API design.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1532/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1072792507,I_kwDOCGYnMM4_8YO7,352,`sqlite-utils insert --extract colname`,9599,open,0,,,4,2021-12-07T00:55:44Z,2022-02-03T22:59:36Z,,OWNER,,"Is there a reason I've not added `--extract` as an option for `sqlite-utils insert` next? There's a `extracts=` option for the various `table.insert()` etc methods - last line in this code block:
https://github.com/simonw/sqlite-utils/blob/213a0ff177f23a35f3b235386366ff132eb879f1/sqlite_utils/db.py#L2483-L2495",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/352/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1087919372,I_kwDOBm6k_c5A2FUM,1578,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key,9599,open,0,,,4,2021-12-23T18:27:59Z,2021-12-24T21:33:19Z,,OWNER,,"Found this while working on https://github.com/simonw/datasette-tiddlywiki
![]()
Then clicking on `/tiddlywiki/tiddlers/%24%3A%2FDefaultTiddlers` returns a 404.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1578/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1131295060,I_kwDOBm6k_c5DbjFU,1634,Update Dockerfile generated by `datasette publish`,9599,open,0,,3268330,4,2022-02-11T00:07:26Z,2022-03-11T17:38:08Z,,OWNER,,"The generated `Dockerfile` currently looks something like this:
```Dockerfile
FROM python:3.8
COPY . /app
WORKDIR /app
ENV DATASETTE_SECRET 'edab49cbc5d5f6f33238f54852037e3fee710821960b73edd2ce743454182ae2'
RUN pip install -U datasette datasette-auth-passwords datasette-tiddlywiki datasette-graphql
RUN datasette inspect fixtures.db other.db --inspect-file inspect-data.json
ENV PORT 8080
EXPOSE 8080
CMD datasette serve --host 0.0.0.0 -i fixtures.db -i other.db --cors --inspect-file inspect-data.json --metadata metadata.json --create --port $PORT /data/*.db
```
This is still on Python 3.8, and it generates a pretty large image compared to the `Dockerfile` used for https://hub.docker.com/datasetteproject/datasette - https://github.com/simonw/datasette/blob/0.60.2/Dockerfile
Here's the code that generates it: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/utils/__init__.py#L389-L400",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1634/reactions"", ""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 2, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1154399841,I_kwDOBm6k_c5Ezr5h,1645,"Sensible `cache-control` headers for static assets, including those served by plugins",697092,open,0,,3268330,4,2022-02-28T18:12:03Z,2022-03-08T02:59:29Z,,NONE,,"## What I'm seeing
With `default_cache_ttl = 86400`, I see the following:
A table view returns `Cache-control: max-age=86400`:
A static asset returns no `Cache-control` header:
## What I expected to see
I expected the static asset to return a `Cache-control` header indicating that this response can be cached.
## Why this matters
I'm productionising a Datasette deployment right now and was looking into putting it behind a Varnish instance. I was surprised to see requests for static assets being served from Datasette rather than Varnish, this is what led me to look more closely at the response headers.
While Datasette serves those static assets pretty quickly, I don't see why Datasette should serve them. By their nature, static assets like images and JS files are very cacheable, so it should be easy to serve them from a cache like Varnish.
(Note that Varnish can easily be configured to override this header, enabling caching for static assets. But it would be better if this override was not necessary.)
## Discussion
It seems clear to me that serving static assets without a `Cache-control` header is not ideal.
I see two options here:
A. Static assets use the same logic as table / SQL views to set the `Cache-control` header based on `default_cache_ttl`.
B. An additional setting for static assets is introduced (`default_static_cache_ttl`, say).",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1645/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1198822563,I_kwDOBm6k_c5HdJSj,1706,"[feature] immutable mode for a directory, not just individual sqlite file",9020979,open,0,,,4,2022-04-10T00:50:57Z,2022-12-09T19:11:40Z,,CONTRIBUTOR,,"## Motivation
- I have a directory of sqlite databases
- I'd like to use immutable mode when opening them for better performance [docs](https://docs.datasette.io/en/0.54/performance.html#immutable-mode)
- Currently using this flag throws the following error
IsADirectoryError: [Errno 21] Is a directory: '/name-of-directory'
## Proposal
Immutable flag works for both single files and directories
datasette -i /folder-of-sqlite-files",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1706/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1237586379,I_kwDOBm6k_c5JxBHL,1742,?_trace=1 fails with datasette-geojson for some reason,9599,open,0,,,4,2022-05-16T19:06:05Z,2022-05-16T19:42:13Z,,OWNER,,view-source:https://calands.datasettes.com/calands/CPAD_2020a_SuperUnits.geojson?_sort=id&id__exact=4&_labels=on&_trace=1 is showing me a blank page.,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1742/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,