`: I thought this was the problem, and it was close, but not quite)
* However `convert()` first wraps the callable by local function [`convert_value()`](https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L2661)
* Consequently `register_function()` sees name `convert_value` for all invocations from `convert()`
* `register_function()` silently ignores registrations using the same name, retaining only the first such registration
There's a mismatch between the comments and the code: https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L404
but actually the existing function is returned/used instead (as the ""registering custom sql functions"" doc I linked above says too). Seems like this can be rectified to match the comment?
## Suggested fix
I think there are four things:
1. The call to `register_function()` from `convert()`should have an explicit `name=` parameter (to continue using `convert_value()` and the progress bar).
2. For functions, this name can be the real function name. (I understand the sqlite api needs a name, and it's nice if those are recognizable names where possible). For lambdas would `'lambda-{uuid}'` or similar be acceptable?
3. `register_function()` really should throw an error on repeated attempts to register a duplicate (function, arity)-pair.
4. A test? I haven't looked at the test framework here but seems this should be testable.
## See also
- #458 ",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/525/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1154399841,I_kwDOBm6k_c5Ezr5h,1645,"Sensible `cache-control` headers for static assets, including those served by plugins",697092,open,0,,3268330,4,2022-02-28T18:12:03Z,2022-03-08T02:59:29Z,,NONE,,"## What I'm seeing
With `default_cache_ttl = 86400`, I see the following:
A table view returns `Cache-control: max-age=86400`:
A static asset returns no `Cache-control` header:
## What I expected to see
I expected the static asset to return a `Cache-control` header indicating that this response can be cached.
## Why this matters
I'm productionising a Datasette deployment right now and was looking into putting it behind a Varnish instance. I was surprised to see requests for static assets being served from Datasette rather than Varnish, this is what led me to look more closely at the response headers.
While Datasette serves those static assets pretty quickly, I don't see why Datasette should serve them. By their nature, static assets like images and JS files are very cacheable, so it should be easy to serve them from a cache like Varnish.
(Note that Varnish can easily be configured to override this header, enabling caching for static assets. But it would be better if this override was not necessary.)
## Discussion
It seems clear to me that serving static assets without a `Cache-control` header is not ideal.
I see two options here:
A. Static assets use the same logic as table / SQL views to set the `Cache-control` header based on `default_cache_ttl`.
B. An additional setting for static assets is introduced (`default_static_cache_ttl`, say).",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1645/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1978603203,I_kwDOCGYnMM517xbD,602,`sqlite-utils transform` removes the `AUTOINCREMENT` keyword,4472046,open,0,,,0,2023-11-06T08:48:43Z,2023-11-06T08:48:43Z,,NONE,,"### Context
We ran into this bug randomly, noticing that deleted `ROWID` would get reused after migrating the DB. Using `transform` to change any column in the table will also unexpectedly strip away the `AUTOINCREMENT` keyword from the primary key definition, even if it was not the transformation target.
### Reproducible example
**Original database**
```sql
$ sqlite3 test.db << EOF
CREATE TABLE mytable (
col1 INTEGER PRIMARY KEY AUTOINCREMENT,
col2 TEXT NOT NULL
)
EOF
$ sqlite3 test.db "".schema mytable""
CREATE TABLE mytable (
col1 INTEGER PRIMARY KEY AUTOINCREMENT,
col2 TEXT NOT NULL
);
```
**Modified database after sqlite-utils**
```sql
$ sqlite-utils transform test.db mytable --rename col2 renamedcol2
$ sqlite3 test.db ""SELECT sql FROM sqlite_master WHERE name = 'mytable';""
CREATE TABLE IF NOT EXISTS ""mytable"" (
[col1] INTEGER PRIMARY KEY,
[renamedcol2] TEXT NOT NULL
);
```",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/602/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
831163537,MDExOlB1bGxSZXF1ZXN0NTkyNTQ4MTAz,1260,Fix: code quality issues,25361949,closed,0,,,2,2021-03-14T13:56:10Z,2021-03-29T00:22:41Z,2021-03-29T00:22:41Z,NONE,simonw/datasette/pulls/1260,"### Description
Hi :wave: I work at [DeepSource](https://deepsource.io), I ran DeepSource analysis on the forked copy of this repo and found some interesting [code quality issues](https://deepsource.io/gh/withshubh/datasette/issues/?category=recommended) in the codebase, opening this PR so you can assess if our platform is right and helpful for you.
### Summary of changes
- Replaced ternary syntax with if expression
- Removed redundant `None` default
- Used `is` to compare type of objects
- Iterated dictionary directly
- Removed unnecessary lambda expression
- Refactored unnecessary `else` / `elif` when `if` block has a `return` statement
- Refactored unnecessary `else` / `elif` when `if` block has a `raise` statement
- Added .deepsource.toml to continuously analyze and detect code quality issues",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1260/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1161937073,I_kwDOBm6k_c5FQcCx,1653,Mechanism to default a table to sorting by multiple columns,9599,open,0,,,2,2022-03-07T21:20:11Z,2022-03-07T21:23:39Z,,OWNER,,"### Discussed in https://github.com/simonw/datasette/discussions/1652
Originally posted by **zaneselvans** March 7, 2022
It's easy to tell datasette to sort tables using a single column, as [described in the docs](https://docs.datasette.io/en/stable/metadata.html#setting-a-default-sort-order):
```yaml
databases:
ferc1:
tables:
f1_edcfu_epda:
sort: created_time
```
But is there some way to tell it to sort using a composite key, like you would in an `ORDER BY` clause instead? For example, the way it's being done **[in this query](https://data.catalyst.coop/ferc1?sql=select%0D%0A++rowid%2C%0D%0A++respondent_id%2C%0D%0A++report_year%2C%0D%0A++spplmnt_num%2C%0D%0A++row_number%2C%0D%0A++row_seq%2C%0D%0A++row_prvlg%2C%0D%0A++acct_num%2C%0D%0A++depr_plnt_base%2C%0D%0A++est_avg_srvce_lf%2C%0D%0A++net_salvage%2C%0D%0A++apply_depr_rate%2C%0D%0A++mrtlty_crv_typ%2C%0D%0A++avg_remaining_lf%2C%0D%0A++report_prd%0D%0Afrom%0D%0A++f1_edcfu_epda%0D%0Awhere%0D%0A++respondent_id+%3D+210%0D%0A++AND+report_year+%3D+2020%0D%0Aorder+by%0D%0A++report_year%2C+report_prd%2C+respondent_id%2C+spplmnt_num%2C+row_number%0D%0Alimit%0D%0A++1000)** on our Datasette?
```sql
SELECT
respondent_id,
report_year,
spplmnt_num,
row_number,
row_seq,
row_prvlg,
acct_num,
depr_plnt_base,
est_avg_srvce_lf,
net_salvage,
apply_depr_rate,
mrtlty_crv_typ,
avg_remaining_lf,
report_prd
FROM
f1_edcfu_epda
WHERE
respondent_id = 210
AND report_year = 2020
ORDER BY
report_year, report_prd, respondent_id, spplmnt_num, row_number
LIMIT
1000
```
The problem here is that by default it's using `rowid` (the SQLite assigned autoincrementing integer key) to order the records, but the table **should** have a natural composite primary key, but the original database that this data is being migrated from doesn't enforce unique primary keys, so there are dupes, and we don't want to drop those rows, and the records are somehow getting jumbled in the database (the `rowid` ordering isn't lined up with the expected ordering based on the composite primary key, though it's close) and this jumbling is confusing to users that expect to see the data ordered based on the natural primary key.
I've tried setting the `sort` metadata parameter to a list of column names, a tuple of column names, a quoted string of comma-separated column names, a quoted string of a tuple of column names...
```yaml
databases:
ferc1:
tables:
f1_edcfu_epda:
sort: ""(report_year, report_prd, respondent_id, spplmnt_num, row_number)""
```
and they all give me server errors like:
```
Cannot sort table by (report_year, report_prd, respondent_id, spplmnt_num, row_number)
```
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1653/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
559964149,MDU6SXNzdWU1NTk5NjQxNDk=,665,Introduce a SQL statement parser in Python,9599,open,0,,,1,2020-02-04T20:36:05Z,2020-02-04T20:36:48Z,,OWNER,,#254 and #653 are both examples of problems that could be solved using a real SQL parser in Python.,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/665/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
323830051,MDU6SXNzdWUzMjM4MzAwNTE=,270,--limit= CLI option for setting limits,9599,closed,0,,,1,2018-05-17T00:14:24Z,2018-05-18T06:19:31Z,2018-05-18T06:16:39Z,OWNER,,"#264 calls for four new datasette limit options, on top of the two existing ones:
* `--max_returned_rows`
* `--sql_time_limit_ms`
These are already clogging up `datasette serve --help` a bit.
How about this syntax instead?
datasette --limit max_returned_rows:100 \
--limit facet_timeout_ms:500 demo.db
Then we can add as many new user over-rideable limits as we like without clogging up `--help` too much - though it would be good to have a way of optionally listings their documentation as well.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/270/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
324652142,MDU6SXNzdWUzMjQ2NTIxNDI=,274,"Rename --limit to --config, add --help-config",9599,closed,0,,,2,2018-05-19T18:57:42Z,2018-05-20T17:04:55Z,2018-05-20T17:04:11Z,OWNER,,"#270 introduced `--limit` but on further thought it should be called `--config` instead.
`--page_size` should becomes `--config default_page_size:1000`
Add `--help-config` to show full help showing all config settings.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/274/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1884330740,PR_kwDOBm6k_c5ZszDF,2174,Use $DATASETTE_INTERNAL in absence of --internal,15178711,open,0,,,3,2023-09-06T16:07:15Z,2023-09-08T00:46:13Z,,CONTRIBUTOR,simonw/datasette/pulls/2174,"#refs 2157, specifically [this comment](https://github.com/simonw/datasette/issues/2157#issuecomment-1700291967)
Passing in `--internal my_internal.db` over and over again can get repetitive.
This PR adds a new configurable env variable `DATASETTE_INTERNAL_DB_PATH`. If it's defined, then it takes place as the path to the internal database. Users can still overwrite this behavior by passing in their own `--internal internal.db` flag.
In draft mode for now, needs tests and documentation.
Side note: Maybe we can have a sections in the docs that lists all the ""configuration environment variables"" that Datasette respects? I did a quick grep and found:
- `DATASETTE_LOAD_PLUGINS`
- `DATASETTE_SECRETS`
----
:books: Documentation preview :books:: https://datasette--2174.org.readthedocs.build/en/2174/
",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2174/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
976399638,MDU6SXNzdWU5NzYzOTk2Mzg=,319,[Enhancement] Please allow 'insert-files' to insert content as text.,66709385,closed,0,,,10,2021-08-22T15:10:46Z,2021-08-24T23:33:45Z,2021-08-24T23:33:44Z,NONE,,"'insert-files' creates BLOB columns for file contents. Transforming the column to TEXT still keep the content as binary. Even though I'm sure there is a transform that can be applied decoding the text it would be great to have a argument to make 'insert-files' to do it as text (with optional text encoding).
The use case is a bunch of htmls (single file) on a directory structure that inserted with this command could be served in Datasette allowing full text search.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/319/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1251710928,I_kwDOBm6k_c5Km5fQ,1751,Add scrollbars to table presentation in default layout,408765,closed,0,,,1,2022-05-28T19:44:57Z,2022-05-28T19:52:17Z,2022-05-28T19:52:17Z,NONE,,"(As you will be able to tell from the terminology I use, I am not a frontend guy, but I hope you will understand.)
When a table is wide and needs horizontal scrolling to see the columns towards the end, the user needs to scroll horizontally. However, since the container for the HTML table (`div` with class `table-wrapper`) isn't limited by the window size, I first need to vertically scroll near to the bottom of the page in order to scroll horizontally. Then I can scroll back up again. This isn't very user friendly. Instead, I think it would make sense to constrain the table's size (when necessary), so that the vertical and horizontal scrollbars either always are visible or at least not far out of reach.
I understand that I could provide my own template and / or CSS, but I think it would probably make sense to adjust the default in this regard.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1751/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
853672224,MDU6SXNzdWU4NTM2NzIyMjQ=,1294,"""You can check out any time you like. But you can never leave!""",192568,open,0,,,0,2021-04-08T17:02:15Z,2021-04-08T18:35:50Z,,CONTRIBUTOR,,"(Feel free to rename this one.)
- The column gear lets you ""Show not-blank rows."" Then it places a parameter in the URL, which a web developer would notice, but a lot of users won't notice, or know to delete it. Would be good to toggle ""Show not-blank rows"" with ""Show all rows."" (Also would be quite helpful to have a ""Show blank rows | Show all rows"" option)
- The column gear lets you ""Sort ascending"" and ""Sort descending"" but then you're stuck with some sort of sorted version thereafter, unless you know to sort the ID column, or to remove the full _sort parameter and its value in the URL. Would be good to offer a ""Remove sort"" option in the gear.
- These requests are in the same camp as: https://github.com/simonw/datasette-vega/issues/36
- I suspect there are other url parameter instances where similar analysis would be helpful, but the three above are the use cases I've run across.
UPDATE:
- It would be helpful to have a ""Previous page"" available for all but the first table page.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1294/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
681575714,MDExOlB1bGxSZXF1ZXN0NDY5OTQ0OTk5,49,"Document the use of --stop_after with favorites, refs #20",370930,closed,0,,,1,2020-08-19T06:10:52Z,2021-08-20T00:02:11Z,2021-08-20T00:02:11Z,CONTRIBUTOR,dogsheep/twitter-to-sqlite/pulls/49,(I discovered this trawling the issues for how to use --since with favorites),206156866,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1465194249,I_kwDOCGYnMM5XVRcJ,514,upsert of new row with check constraints fails,193185,closed,0,,,5,2022-11-26T16:12:23Z,2023-05-08T21:50:52Z,2023-05-08T21:50:51Z,NONE,,"(I originally opened this in https://github.com/simonw/datasette-insert/issues/20, but I see that that library depends on sqlite-utils)
In the case of a new row, upsert first adds the row, specifying only its pkeys: https://github.com/simonw/sqlite-utils/blob/965ca0d5f5bffe06cc02cd7741344d1ddddf9d56/sqlite_utils/db.py#L2783-L2787
This means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/514/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
988556488,MDU6SXNzdWU5ODg1NTY0ODg=,1459,suggestion: allow `datasette --open` to take a relative URL,51016,open,0,,,1,2021-09-05T17:17:07Z,2021-09-05T19:59:15Z,,CONTRIBUTOR,,"(soft suggestion because I'm not sure I'm using datasette right yet)
Over at https://github.com/ctb/2021-sourmash-datasette, I'm playing around with datasette, and I'm creating some static pages to send people to the right facets. There may well be better ways of achieving this end goal, and I will find out if so, I'm sure!
But regardless I think it might be neat to support an option to allow `-o/--open` to take a relative URL, that then gets appended to the hostname and port. This would let me improve my documentation. I don't see any downsides, either, but 🤷 there may well be some :)
Happy to dig in and provide a PR if it's of interest. I'm not sure off the top of my head how to support an optional value to a parameter in argparse - the current `-o` behavior is kinda nice so it'd be suboptimal to require a url for `-o`. Maybe `--open-url=` or something would work?
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1459/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
322591993,MDExOlB1bGxSZXF1ZXN0MTg3NjY4ODkw,257,Refactor views,9599,closed,0,,,5,2018-05-13T13:00:50Z,2018-05-14T03:04:25Z,2018-05-14T03:04:24Z,OWNER,simonw/datasette/pulls/257,"* Split out view classes from main `app.py`
* Run [black](https://github.com/ambv/black) against resulting code to apply opinionated source code formatting
* Run [isort](https://github.com/timothycrosley/isort) to re-order my imports
Refs #256 ",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/257/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1393330070,PR_kwDODD6af84__DNJ,14,Photo links,6782721,open,0,,,0,2022-10-01T09:44:15Z,2022-11-18T17:10:49Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/swarm-to-sqlite/pulls/14,"* add to `checkin_details` view new column for a calculated photo links
* supported multiple links split by newline
* create `events` table if there's no events in the history to avoid SQL errors
Fixes #9.",205429375,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/14/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1538197093,I_kwDOBm6k_c5brwZl,1995,foreign_keys error 500,137183,open,0,,,0,2023-01-18T15:27:36Z,2023-01-18T16:44:01Z,,NONE,,"**Error 500 expected string or bytes-like object**
[espial-new.sqlite3.zip](https://github.com/simonw/datasette/files/10447965/espial-new.sqlite3.zip)
run `datasette espial-new.sqlite3` & click on any table other than `User`
```
/home/jon/.local/lib/python3.10/site-packages/datasette/app.py:814 in │
│ expand_foreign_keys │
│ │
│ 811 │ │ │ from {other_table} │
│ 812 │ │ │ where {other_column} in ({placeholders}) │
│ 813 │ │ """""".format( │
│ ❱ 814 │ │ │ other_column=escape_sqlite(fk[""other_column""]), │
│ 815 │ │ │ label_column=escape_sqlite(label_column), │
│ 816 │ │ │ other_table=escape_sqlite(fk[""other_table""]), │
│ 817 │ │ │ placeholders="", "".join([""?""] * len(set(values))), │
│ │
│ ╭───────────────────────────── locals ──────────────────────────────╮ │
│ │ column = 'user_id' │ │
│ │ database = 'espial-new' │ │
│ │ db = │ │
│ │ fk = { │ │
│ │ │ 'column': 'user_id', │ │
│ │ │ 'other_table': 'user', │ │
│ │ │ 'other_column': None │ │
│ │ } │ │
│ │ foreign_keys = [ │ │
│ │ │ { │ │
│ │ │ │ 'column': 'user_id', │ │
│ │ │ │ 'other_table': 'user', │ │
│ │ │ │ 'other_column': None │ │
│ │ │ } │ │
│ │ ] │ │
│ │ label_column = 'name' │ │
│ │ labeled_fks = {} │ │
│ │ self = │ │
│ │ table = 'bookmark' │ │
│ │ values = [] │ │
│ ╰───────────────────────────────────────────────────────────────────╯ │
│ │
│ /home/jon/.local/lib/python3.10/site-packages/datasette/utils/__init__.py:346 │
│ in escape_sqlite │
│ │
│ 343 │
│ 344 │
│ 345 def escape_sqlite(s): │
│ ❱ 346 │ if _boring_keyword_re.match(s) and (s.lower() not in reserved_words) │
│ 347 │ │ return s │
│ 348 │ else: │
│ 349 │ │ return f""[{s}]"" │
│ │
│ ╭─ locals ─╮ │
│ │ s = None │ │
│ ╰──────────╯ │
╰─────────────────────────────────────────────────────────────────────────────────╯
TypeError: expected string or bytes-like object
Traceback (most recent call last):
File ""/home/jon/.local/lib/python3.10/site-packages/datasette/app.py"", line 1354, in route_path
response = await view(request, send)
File ""/home/jon/.local/lib/python3.10/site-packages/datasette/views/base.py"", line 134, in view
return await self.dispatch_request(request)
File ""/home/jon/.local/lib/python3.10/site-packages/datasette/views/base.py"", line 91, in dispatch_request
return await handler(request)
File ""/home/jon/.local/lib/python3.10/site-packages/datasette/views/base.py"", line 361, in get
response_or_template_contexts = await self.data(request, **data_kwargs)
File ""/home/jon/.local/lib/python3.10/site-packages/datasette/views/table.py"", line 158, in data
return await self._data_traced(request, default_labels, _next, _size)
File ""/home/jon/.local/lib/python3.10/site-packages/datasette/views/table.py"", line 603, in _data_traced
await self.ds.expand_foreign_keys(
File ""/home/jon/.local/lib/python3.10/site-packages/datasette/app.py"", line 814, in expand_foreign_keys
other_column=escape_sqlite(fk[""other_column""]),
File ""/home/jon/.local/lib/python3.10/site-packages/datasette/utils/__init__.py"", line 346, in escape_sqlite
if _boring_keyword_re.match(s) and (s.lower() not in reserved_words):
TypeError: expected string or bytes-like object
INFO: 127.0.0.1:38574 - ""GET /espial-new/bookmark HTTP/1.1"" 500 Internal Server Error
INFO: 127.0.0.1:38574 - ""GET /-/static/app.css?d59929 HTTP/1.1"" 200 OK
```
Schema:
```
CREATE TABLE IF NOT EXISTS ""user""
(
""id"" INTEGER PRIMARY KEY,
""name"" VARCHAR NOT NULL,
""password_hash"" VARCHAR NOT NULL,
""api_token"" VARCHAR NULL,
""private_default"" BOOLEAN NOT NULL,
""archive_default"" BOOLEAN NOT NULL,
""privacy_lock"" BOOLEAN NOT NULL,
CONSTRAINT ""unique_user_name"" UNIQUE (""name"")
);
CREATE TABLE IF NOT EXISTS ""bookmark""
(
""id"" INTEGER PRIMARY KEY,
""user_id"" INTEGER NOT NULL REFERENCES ""user"" ON DELETE RESTRICT ON UPDATE RESTRICT,
""slug"" VARCHAR NOT NULL DEFAULT (Lower(Hex(Randomblob(6)))),
""href"" VARCHAR NOT NULL,
""description"" VARCHAR NOT NULL,
""extended"" VARCHAR NOT NULL,
""time"" TIMESTAMP NOT NULL,
""shared"" BOOLEAN NOT NULL,
""to_read"" BOOLEAN NOT NULL,
""selected"" BOOLEAN NOT NULL,
""archive_href"" VARCHAR NULL,
CONSTRAINT ""unique_user_href"" UNIQUE (""user_id"", ""href""),
CONSTRAINT ""unique_user_slug"" UNIQUE (""user_id"", ""slug"")
);
CREATE TABLE IF NOT EXISTS ""bookmark_tag""
(
""id"" INTEGER PRIMARY KEY,
""user_id"" INTEGER NOT NULL REFERENCES ""user"" ON DELETE RESTRICT ON UPDATE RESTRICT,
""tag"" VARCHAR NOT NULL,
""bookmark_id"" INTEGER NOT NULL REFERENCES ""bookmark"" ON DELETE RESTRICT ON UPDATE RESTRICT,
""seq"" INTEGER NOT NULL,
CONSTRAINT ""unique_user_tag_bookmark_id"" UNIQUE (""user_id"", ""tag"", ""bookmark_id""),
CONSTRAINT ""unique_user_bookmark_id_tag_seq"" UNIQUE (""user_id"", ""bookmark_id"", ""tag"", ""seq"")
);
CREATE TABLE IF NOT EXISTS ""note""
(
""id"" INTEGER PRIMARY KEY,
""user_id"" INTEGER NOT NULL REFERENCES ""user"" ON DELETE RESTRICT ON UPDATE RESTRICT,
""slug"" VARCHAR NOT NULL DEFAULT (Lower(Hex(Randomblob(10)))),
""length"" INTEGER NOT NULL,
""title"" VARCHAR NOT NULL,
""text"" VARCHAR NOT NULL,
""is_markdown"" BOOLEAN NOT NULL,
""shared"" BOOLEAN NOT NULL DEFAULT false,
""created"" TIMESTAMP NOT NULL,
""updated"" TIMESTAMP NOT NULL
);
CREATE INDEX idx_bookmark_time ON bookmark (user_id, time DESC);
CREATE INDEX idx_bookmark_tag_bookmark_id ON bookmark_tag (bookmark_id, id, tag, seq);
CREATE INDEX idx_note_user_created ON note (user_id, created DESC);
```
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1995/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
904598267,MDExOlB1bGxSZXF1ZXN0NjU1NzQxNDI4,1348,DRAFT: add test and scan for docker images,10801138,open,0,,,2,2021-05-28T03:02:12Z,2021-05-28T03:06:16Z,,CONTRIBUTOR,simonw/datasette/pulls/1348,"**NOTE: I don't think this PR is ready, since the arm/v6 and arm/v7 images are failing pytest due to missing dependencies (gcc and friends). But it's pretty close.**
Closes https://github.com/simonw/datasette/issues/1344 . Using a build-matrix for the platforms and [this test](https://github.com/simonw/datasette/issues/1344#issuecomment-849820019), we test all the platforms in parallel. I also threw in container scanning.
### Switch `pip install` to use either tags or commit shas
Notably! This also [changes the Dockerfile](https://github.com/blairdrummond/datasette/blob/7fe5315d68e04fce64b5bebf4e2d7feec44f8546/Dockerfile#L20) so that it accepts tags or commit-shas.
```
# It's backwards compatible with tags, but also lets you use shas
root@712071df17af:/# pip install git+git://github.com/simonw/datasette.git@0.56
Collecting git+git://github.com/simonw/datasette.git@0.56
Cloning git://github.com/simonw/datasette.git (to revision 0.56) to /tmp/pip-req-build-u6dhm945
Running command git clone -q git://github.com/simonw/datasette.git /tmp/pip-req-build-u6dhm945
Running command git checkout -q af5a7f1c09f6a902bb2a25e8edf39c7034d2e5de
Collecting Jinja2<2.12.0,>=2.10.3
Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
```
This lets you build the containers in CI every push for testing, which maybe resolves [this problem](https://github.com/simonw/datasette/issues/1272#issuecomment-808648974)?
# Workflow run example
You can see the results in my workflow [here](https://github.com/blairdrummond/datasette/pull/2/checks?check_run_id=2690570717). The commit history is different because I squashed this branch, also in the testing branch I had to change `github.com/simonw` to `github.com/blairdrummond` for the CI to pick up my git_sha.
## Why did the builds fail?
**NOTE:** The results of all the tests fail, but for different reasons! A few fail to install Rust, the amd64 passes the tests (phew!) but has critical CVEs which fail the container scan, the Arm/v6 and Arm/v7 seem to fail to install the test dependencies due to missing programs like `gcc`. (`gcc` is not sufficient though, as [this run](https://github.com/blairdrummond/datasette/pull/3/checks?check_run_id=2690672982) indicates) ",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1348/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
807174161,MDU6SXNzdWU4MDcxNzQxNjE=,227,Error reading csv files with large column data,295329,closed,0,,,4,2021-02-12T11:51:47Z,2021-02-16T11:48:03Z,2021-02-14T21:17:19Z,NONE,,"*Feel free to close this issue - I mostly added it for reference for future folks that run into this :)*
I have a CSV file with one column that has very long strings. When i try to import this file via the `insert` command I get the following error:
```
sqlite-utils insert database.db table_name file_with_large_column.csv
Traceback (most recent call last):
File ""/usr/local/bin/sqlite-utils"", line 10, in
sys.exit(cli())
File ""/usr/local/lib/python3.7/site-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/usr/local/lib/python3.7/site-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/usr/local/lib/python3.7/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/usr/local/lib/python3.7/site-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/usr/local/lib/python3.7/site-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py"", line 774, in insert
default=default,
File ""/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py"", line 705, in insert_upsert_implementation
docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs
File ""/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1852, in insert_all
first_record = next(records)
File ""/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py"", line 703, in
docs = (decode_base64_values(doc) for doc in docs)
File ""/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py"", line 681, in
docs = (dict(zip(headers, row)) for row in reader)
_csv.Error: field larger than field limit (131072)
```
Built with the docker image `datasetteproject/datasette:0.54` with the following versions:
```
# sqlite-utils --version
sqlite-utils, version 3.4.1
# datasette --version
datasette, version 0.54
```
It appears this is a [known issue](https://stackoverflow.com/a/54517228/2761423) reading in csv files in python and [doesn't look to be modifiable](https://github.com/python/cpython/blob/ea46579067fd2d4e164d6605719ffec690c4d621/Modules/_csv.c#L1685) through system / env vars (i may be very wrong on this).
Noting that using sqlite3 `import` command work without error (not using the python csv reader)
```
sqlite3 database.db
sqlite> .mode csv
sqlite> .import file_with_large_column.csv table_name
```
Sadly I couldn't see an easy way around this while using the cli as it appears this value needs to be changed in python code. FWIW I've switched to using https://datasette.io/tools/csvs-to-sqlite for importing csv data and it's working well.
Finally, I'm loving https://datasette.io/ thank you very much for an amazing tool and data ecosytem 🙇♀️ ",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/227/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1114147905,I_kwDOBm6k_c5CaIxB,1612,Move canned queries closer to the SQL input area,639012,closed,0,,3268330,5,2022-01-25T17:06:39Z,2022-03-19T04:04:49Z,2022-01-25T18:34:21Z,CONTRIBUTOR,,"*Original title: Consider placing example queries above the sql input?*
Hi! Have been enjoying deploying ad hoc datasettes for collaborators to pick over!
I keep finding myself manually ""fixing"" the database.html template so that the ""example queries"" (canned queries) appear directly *over* the sql box? So they are sorta more a suggestion for collaborators who aren't inclined to write their own queries?
My sense is any time I go to the trouble of writing canned queries my users should see 'em?
(( I have also considered a client-side reactive-ish option where selecting a query just places the raw SQL in the box and doesn't execute it, but this seems to end up being an inconvenience, rather than a teaching tool. ))
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1612/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
860625833,MDU6SXNzdWU4NjA2MjU4MzM=,1300,Make row available to `render_cell` plugin hook,3243482,closed,0,,,5,2021-04-18T10:14:37Z,2022-07-07T16:34:05Z,2022-07-07T16:31:22Z,CONTRIBUTOR,,"*Original title: **Generating URL for a row inside `render_cell` hook***
Hey,
I am using Datasette to view a database that contains video metadata. It has BLOB columns that contain video thumbnails in JPG format (around 100-500KB per row).
I've registered an output formatter that extends `datasette.blob_renderer.render_blob` function and serves the column with `image/jpeg` content type.
```python
from datasette.blob_renderer import render_blob
async def render_jpg(datasette, database, rows, columns, request, table, view_name):
response = await render_blob(datasette, database, rows, columns, request, table, view_name)
response.content_type = ""image/jpeg""
response.headers[""Content-Disposition""] = f'inline; filename=""image.jpg""'
return response
@hookimpl
def register_output_renderer():
return {
""extension"": ""jpg"",
""render"": render_jpg,
""can_render"": lambda: True,
}
```
This works well. I can visit `http://localhost:8001/mydb/videos/1.jpg?_blob_column=thumbnail` and view the image.
I want to display the image directly with an `![]()
` tag (lazy-loaded of course). So, I need a URL, because embedding base64 would increase the page size too much (each image > 100KB).
Datasette generates a link with `.blob` extension for blob columns. It does this by calling `datasette.urls.row_blob`
https://github.com/simonw/datasette/blob/7a2ed9f8a119e220b66d67c7b9e07cbab47b1196/datasette/views/table.py#L169-L179
But I have no way of getting the row inside the `render_cell` hook.
```python
@hookimpl
def render_cell(value, column, table, database, datasette):
if isinstance(value, bytes) and imghdr.what(None, value):
# generate url
return '$renderedLink'
```
Any pointers?",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1300/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1126692066,I_kwDOCGYnMM5DJ_Ti,403,Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`,536941,closed,0,,,4,2022-02-08T01:39:40Z,2022-02-09T04:22:43Z,2022-02-08T19:33:59Z,CONTRIBUTOR,,"*Original title: Add option for adding a new, serial, primary key*
sometimes we have tables that don't have primary keys, but ought to have them. we *can* use rowid for that, but it would often be nicer to have an explicit primary key. using the current value of rowid would be fine.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/403/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1087913724,I_kwDOBm6k_c5A2D78,1577,Drop support for Python 3.6,9599,closed,0,,3268330,6,2021-12-23T18:17:03Z,2022-01-25T23:30:03Z,2022-01-20T04:31:41Z,OWNER,,"*Original title: Decide when to drop support for Python 3.6*
> `context_vars` can solve this but they were introduced in Python 3.7: https://www.python.org/dev/peps/pep-0567/
>
> Python 3.6 support ends in a few days time, and it looks like Glitch has updated to 3.7 now - so maybe I can get away with Datasette needing 3.7 these days?
>
> Tweeted about that here: https://twitter.com/simonw/status/1473761478155010048
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1576#issuecomment-999878907_",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1577/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
816560819,MDU6SXNzdWU4MTY1NjA4MTk=,240,table.pks_and_rows_where() method returning primary keys along with the rows,9599,closed,0,,,7,2021-02-25T15:49:28Z,2021-02-25T16:39:23Z,2021-02-25T16:28:23Z,OWNER,,"*Original title: Easier way to update a row returned from .rows*
Here's a surprisingly hard problem I ran into while trying to implement #239 - given a row returned by `db[table].rows` how can you update that row?
The problem is that the `db[table].update(...)` method requires a primary key. But if you have a row from the `db[table].rows` iterator it might not even contain the primary key - provided the table is a `rowid` table.
Instead, currently, you need to introspect the table and, if `rowid` is a primary key, explicitly include that in the `select=` argument to `table.rows_where(...)` - otherwise it will not be returned.
A utility mechanism to make this easier would be very welcome.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/240/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1077620955,I_kwDOBm6k_c5AOzDb,1549,Redesign CSV export to improve usability,536941,open,0,,3268330,5,2021-12-11T19:02:12Z,2022-04-04T11:17:13Z,,CONTRIBUTOR,,"*Original title: Set content type for CSV so that browsers will attempt to download instead opening in the browser*
Right now, if the user clicks on the CSV related to a table or a query, the response header for the content type is
""content-type: text/plain; charset=utf-8""
Most browsers will try to open a file with this content-type in the browser.
This is not what most people want to do, and lots of folks don't know that if they want to download the CSV and open it in the a spreadsheet program they next need to save the page through their browser.
It would be great if the response header could be something like
```
'Content-type: text/csv');
'Content-disposition: attachment;filename=MyVerySpecial.csv');
```
which would lead browsers to open a download dialog.
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1549/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1250629388,I_kwDOCGYnMM5KixcM,440,CSV files with too many values in a row cause errors,4068,closed,0,,,20,2022-05-27T10:54:44Z,2022-06-14T22:23:01Z,2022-06-14T20:12:46Z,NONE,,"*Original title: csv.DictReader can have None as key*
In some cases, `csv.DictReader` can have `None` as key for unnamed columns, and a list of values as value.
`sqlite_utils.utils.rows_from_file` cannot handle that:
```python
url=""https://artsdatabanken.no/Fab2018/api/export/csv""
db = sqlite_utils.Database("":memory"")
with urlopen(url) as fab:
reader, _ = sqlite_utils.utils.rows_from_file(fab, encoding=""utf-16le"")
db[""fab2018""].insert_all(reader, pk=""Id"")
```
Result:
```
Traceback (most recent call last):
File """", line 3, in
File ""/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py"", line 2924, in insert_all
chunk = list(chunk)
File ""/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py"", line 3454, in fix_square_braces
if any(""["" in key or ""]"" in key for key in record.keys()):
File ""/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py"", line 3454, in
if any(""["" in key or ""]"" in key for key in record.keys()):
TypeError: argument of type 'NoneType' is not iterable
```
Code:
https://github.com/simonw/sqlite-utils/blob/59be60c471fd7a2c4be7f75e8911163e618ff5ca/sqlite_utils/db.py#L3454
`sqlite-utils insert` from command line is not affected by this issue.",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/440/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
656959584,MDU6SXNzdWU2NTY5NTk1ODQ=,893,pip3 install datasette not serving static on linuxbrew.,44167,closed,0,,,1,2020-07-14T23:33:38Z,2021-06-02T04:29:56Z,2021-06-02T04:29:56Z,NONE,,"*This error wasn't thrown*
```
Traceback (most recent call last):
File ""/home/linuxbrew/.linuxbrew/opt/python@3.8/lib/python3.8/site-packages/datasette/utils/asgi.py"", line 289, in inner_static
full_path.relative_to(root_path)
File ""/home/linuxbrew/.linuxbrew/opt/python@3.8/lib/python3.8/pathlib.py"", line 904, in relative_to
raise ValueError(""{!r} does not start with {!r}""
ValueError: '/home/linuxbrew/.linuxbrew/lib/python3.8/site-packages/datasette/static/app.css' does not start with '/home/linuxbrew/.linuxbrew/opt/python@3.8/lib/python3.8/site-packages/datasette/static'
```
Linuxbrew install python@3.8 with symbolic links when You call the full_path.relative_to(root_path) throw ValueError. This happened when you install from pip3
when you install with python3 setup.py develop , works good.
Well at the end the static wasn't serving.
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/893/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1822939274,I_kwDOBm6k_c5sp9iK,2113,Implement and document extras for the new query view page,9599,open,0,,8755003,3,2023-07-26T18:24:01Z,2023-08-09T17:35:22Z,,OWNER,,- #2109 ,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2113/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1822940263,I_kwDOBm6k_c5sp9xn,2114,Implement canned queries against new query JSON work,9599,closed,0,,9700784,3,2023-07-26T18:24:50Z,2023-08-09T15:26:58Z,2023-08-09T15:26:57Z,OWNER,,- #2109 ,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2114/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1822940964,I_kwDOBm6k_c5sp98k,2115,Ensure all tests pass against new query view JSON,9599,closed,0,,9700784,0,2023-07-26T18:25:20Z,2023-08-08T02:01:39Z,2023-08-08T02:01:38Z,OWNER,,- #2109 ,107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2115/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1874327336,PR_kwDOBm6k_c5ZLMSe,2165,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins,9599,closed,0,,,6,2023-08-30T20:33:30Z,2023-08-30T22:12:25Z,2023-08-30T22:12:25Z,OWNER,simonw/datasette/pulls/2165,"- #2164
TODO:
- [x] Automated tests
- [ ] Documentation
- [x] Make sure `DATASETTE_LOAD_PLUGINS=''` works for loading zero plugins",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2165/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
924992318,MDU6SXNzdWU5MjQ5OTIzMTg=,281,Mechanism for explicitly stating CSV or JSON or TSV for sqlite-utils memory,9599,closed,0,,,1,2021-06-18T15:04:53Z,2021-06-19T03:11:59Z,2021-06-19T03:11:59Z,OWNER,,"- Implement `filename.json:json` and `-:nl` and suchlike options for specifying the format rather than guessing it - see https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861985944
Follows #272",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/281/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
731827081,MDExOlB1bGxSZXF1ZXN0NTExODY4MTUz,1060,New explicit versioning mechanism,9599,closed,0,,6026070,1,2020-10-28T22:14:55Z,2020-10-29T03:38:17Z,2020-10-29T03:38:16Z,OWNER,simonw/datasette/pulls/1060,"- Remove all references to versioneer
- Re-implement versioning to use a static string baked into the repo
- Ensure that string is output by `datasette --version` and `/-/versions`
Refs #1054",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1060/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
924990677,MDU6SXNzdWU5MjQ5OTA2Nzc=,279,sqlite-utils memory should handle TSV and JSON in addition to CSV,9599,closed,0,,,7,2021-06-18T15:02:54Z,2021-06-19T03:11:59Z,2021-06-19T03:11:59Z,OWNER,,"- Use sniff to detect CSV or TSV (if `:tsv` or `:csv` was not specified) and delimiters
Follow-on from #272",140912432,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/279/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
276704327,MDU6SXNzdWUyNzY3MDQzMjc=,150,_group_count= feature improvements,9599,closed,0,,,3,2017-11-24T22:06:18Z,2018-05-28T16:41:28Z,2018-05-28T16:41:28Z,OWNER,,"- [ ] The ""apply filters"" form should keep you on the _group_count= page
- [ ] Foreign key references should be expand
- [ ] Page title should reflect the view you are on",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/150/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
268262480,MDU6SXNzdWUyNjgyNjI0ODA=,36,"date, year, month and day querystring lookups",9599,closed,0,,,3,2017-10-25T04:23:45Z,2018-05-28T17:30:53Z,2018-05-28T17:30:53Z,OWNER,,"- [ ] `?timestamp___date=2017-07-17` - return every item where the timestamp falls on that date
- [ ] `?timestamp___year=2017` - return every item where the timestamp falls within 2017
- [ ] `?timestamp___month=1` - return every item where the month component is January
- [ ] `?timestamp___day=10` - return every item where the day-of-the-month component is 10
Follow on from #23 ",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/36/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
602533539,MDU6SXNzdWU2MDI1MzM1Mzk=,4,Upload all my photos to a secure S3 bucket,9599,closed,0,,5324096,14,2020-04-18T19:24:50Z,2020-04-18T21:58:11Z,2020-04-18T21:57:13Z,MEMBER,,"- [x] Create a bucket with bucket credentials
- [x] Programmatically upload some recent photos to it (from a notebook)
- [x] Turn this into a script",256834907,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
601265023,MDU6SXNzdWU2MDEyNjUwMjM=,25,Improvements to demo instance,9599,closed,0,,,1,2020-04-16T17:26:55Z,2020-04-16T18:07:12Z,2020-04-16T18:07:12Z,MEMBER,,- [x] Demo should pull issue-comments as well,207052882,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
732634375,MDExOlB1bGxSZXF1ZXN0NTEyNTQ1MzY0,1061,.blob output renderer,9599,closed,0,,6026070,4,2020-10-29T20:25:08Z,2020-10-29T22:01:40Z,2020-10-29T22:01:39Z,OWNER,simonw/datasette/pulls/1061,"- [x] Remove the `/-/...blob/...` route I added in #1040 in place of the new `.blob` renderer URLs
- [x] Link to new `.blob` download links on the arbitrary query page (using `_blob_hash=...`) - plus tests for this
Closes #1050, Closes #1051",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1061/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
463492395,MDExOlB1bGxSZXF1ZXN0MjkzOTYyNDA1,533,"Support cleaner custom templates for rows and tables, closes #521",9599,closed,0,,,1,2019-07-03T00:40:18Z,2019-07-03T03:23:06Z,2019-07-03T03:23:06Z,OWNER,simonw/datasette/pulls/533,"- [x] Rename `_rows_and_columns.html` to `_table.html`
- [x] Unit test
- [x] Documentation",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/533/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
467928674,MDExOlB1bGxSZXF1ZXN0Mjk3NDU5Nzk3,40,.get() method plus support for compound primary keys,9599,closed,0,,,1,2019-07-15T03:43:13Z,2019-07-15T04:28:57Z,2019-07-15T04:28:52Z,OWNER,simonw/sqlite-utils/pulls/40,"- [x] Tests for the `NotFoundError` exception
- [x] Documentation for `.get()` method
- [x] Support `--pk` multiple times to define CLI compound primary keys
- [x] Documentation for compound primary keys",140912432,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/40/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
732685643,MDU6SXNzdWU3MzI2ODU2NDM=,1063,.csv should link to .blob downloads,9599,closed,0,,6026070,3,2020-10-29T21:45:58Z,2021-06-17T18:12:30Z,2020-10-29T22:47:45Z,OWNER,,"- [x] Update `.csv` output to link to these things (and get that `xfail` test to pass)
- ~~Add a `.csv?_blob_base64=1` argument that causes them to be output in base64 in the CSV~~
> Moving the CSV work to a separate ticket.
_Originally posted by @simonw in https://github.com/simonw/datasette/pull/1061#issuecomment-719042601_",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1063/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
679779797,MDU6SXNzdWU2Nzk3Nzk3OTc=,939,extra_ plugin hooks should take the same arguments,9599,closed,0,,,6,2020-08-16T16:04:54Z,2020-08-16T18:25:05Z,2020-08-16T16:50:29Z,OWNER,,"- [x] `extra_css_urls(template, database, table, datasette)`
- [x] `extra_js_urls(template, database, table, datasette)`
- [x] `extra_body_script(template, database, table, view_name, datasette)`
- [x] `extra_template_vars(template, database, table, view_name, request, datasette)`
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/938#issuecomment-674544691_",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/939/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
476436920,MDExOlB1bGxSZXF1ZXN0MzAzOTkwNjgz,53,Work in progress: m2m() method for creating many-to-many records,9599,closed,0,,,0,2019-08-03T10:03:56Z,2019-08-04T03:38:10Z,2019-08-04T03:37:33Z,OWNER,simonw/sqlite-utils/pulls/53,"- [x] `table.insert({""name"": ""Barry""}).m2m(""tags"", lookup={""tag"": ""Coworker""})`
- [x] Explicit table name `.m2m(""humans"", ..., m2m_table=""relationships"")`
- [x] Automatically use an existing m2m table if a single obvious candidate exists (a table with two foreign keys in the correct directions)
- [x] Require the explicit `m2m_table=` argument if multiple candidates for the m2m table exist
- [x] Documentation
Refs #23",140912432,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/53/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
565041624,MDU6SXNzdWU1NjUwNDE2MjQ=,671,"datasette.add_database(name, db) and datasette.remove_database(name) methods",9599,closed,0,,,1,2020-02-14T01:05:48Z,2020-02-14T01:30:35Z,2020-02-14T01:30:30Z,OWNER,,"- `datasette.add_database(name, db)` - adds a new named database to the list of connected databases. `db` will be a `Database()` object, which may prove useful in the future for things like #670 and could also allow some plugins to provide in-memory SQLite databases.
- `datasette.remove_database(name)`
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/417#issuecomment-586047995_",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/671/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1432012302,I_kwDOBm6k_c5VWsYO,1877,Refactor and tidy up final write API code,9599,closed,0,,,1,2022-11-01T20:00:11Z,2022-11-29T19:44:16Z,2022-11-29T19:44:07Z,OWNER,,"- `views/table.py` has got a bit too big - I think the write classes should be pulled out into a separate module.
- [x] There's duplicate logic for deciding if the table and database exist and checking permissions",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1877/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1884499674,PR_kwDODFE5qs5ZtYMc,13,"use poetry for packages, asdf for versioning, and gh actions for ci",150855,open,0,,,0,2023-09-06T17:59:16Z,2023-09-06T17:59:16Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/google-takeout-to-sqlite/pulls/13,"- build: use poetry for package management, asdf for python version
- build: cleanup poetry config, add keywords, ignore dist
- ci: migrate circleci to gh actions
- fix: dup method definition
",206649770,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/13/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
773913793,MDExOlB1bGxSZXF1ZXN0NTQ0OTIzNDM3,1158,Modernize code to Python 3.6+,6774676,closed,0,,6346396,4,2020-12-23T16:21:38Z,2021-01-24T21:20:50Z,2020-12-23T17:04:32Z,CONTRIBUTOR,simonw/datasette/pulls/1158,"- compact dict and set building
- remove redundant parentheses
- simplify chained conditions
- change method name to lowercase
- use triple double quotes for docstrings
please feel free to accept/reject any of these independent commits",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1158/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
2001006157,PR_kwDOCGYnMM5f2OZC,604,Add more STRICT table support,16437338,closed,0,,,4,2023-11-19T19:38:53Z,2023-12-08T05:17:20Z,2023-12-08T05:05:27Z,CONTRIBUTOR,simonw/sqlite-utils/pulls/604,"- https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776
Make `table.transform()` preserve STRICT mode.
----
:books: Documentation preview :books:: https://sqlite-utils--604.org.readthedocs.build/en/604/
",140912432,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/604/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1586980089,PR_kwDOBm6k_c5KF-by,2026,Avoid repeating primary key columns if included in _col args,8513,open,0,,,0,2023-02-16T04:16:25Z,2023-02-16T04:16:41Z,,FIRST_TIME_CONTRIBUTOR,simonw/datasette/pulls/2026,"...while maintaining given order.
Fixes #1975 (if I'm understanding correctly).
----
:books: Documentation preview :books:: https://datasette--2026.org.readthedocs.build/en/2026/
",107914493,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/2026/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
608613033,MDU6SXNzdWU2MDg2MTMwMzM=,745,Extract the hash-URL mechanism out into a plugin,9599,closed,0,,,2,2020-04-28T21:00:38Z,2020-10-23T19:47:18Z,2020-10-23T19:47:10Z,OWNER,,"0.28 in May 2019 made this feature not-the-default: https://datasette.readthedocs.io/en/stable/changelog.html#v0-28 - see #418
I've not felt the need to use it myself since. I think I should move it into a plugin.",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/745/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
860734722,MDU6SXNzdWU4NjA3MzQ3MjI=,1302,Fix disappearing facets,192568,open,0,,,0,2021-04-18T18:42:33Z,2021-04-20T07:40:15Z,,CONTRIBUTOR,,"1. Clone https://github.com/mroswell/list-N
2. Run `datasette disinfectants.db -o`
3. Select the `Safer_or_Toxic` facet.
4. Select `Toxic`.
5. Close out the `Safer_or_Toxic` facet.
6. Examine `Suggested facets` list. `Safer_or_Toxic` is GONE.
7. Try some other facets. When you select an element, and then close the list, in some cases, the facet properly returns to the `Suggested facet` list... Arrays and dates properly return to the list, but fields with strings don't return to the list.
Since my site is devoted to whether disinfectants are Safer or Toxic, having the suggested facet disappear from the suggested facet list is very confusing* to end-users. This, along with a few other issues, unfortunately proved beyond my own programming ability to address. So I hired a Senior-level developer to address a number of issues, including this disappearing act.
8. Open a new terminal. Run `datasette disinfectants.db -m metadata.json --static static:static/ --template-dir templates/ --plugins-dir plugins/ -p 8001 -o`
9. Repeat steps 3-6, but this time, the Safer_or_Toxic facet returns to the list (and the related URL parameters are removed).
I'm not sure how to do a pull request for this, because the plugin contains other functionality that goes beyond this bug. I wanted the facets sorted in a certain order (both in the suggested facet list, and the detail lists) (... the detail lists were hopping around all over the place before...) I wanted the duplicate facets removed (leaving only the one where you can facet by individual item in an array.) I wanted the arrays to be presented in a prettier fashion (I did that in the template... That could be moved over to the plugin at some point)
I'm thinking it'll be very helpful if applicable parts of my project's plugin (sort_suggested_facets_plugin.py) will be able to be incorporated back into datasette, but I leave that to you to consider.
(* The disappearing facet bug was especially confusing because I'm removing the filters and sql from the table page, at the request of the organization. The filters and sql detail created a lot of confusion for end users who try to find disinfectants used by Hospitals, for instance, as an '=' won't find them, since they are part of the Use_site array.) My disappearing-facet confusion was documented in my own issue: https://github.com/mroswell/list-N/issues/57 (addressed by the plugin). Other facet-related issues here: https://github.com/mroswell/list-N/issues/54 (addressed by the plugin); https://github.com/mroswell/list-N/issues/15 (addressed by template); https://github.com/mroswell/list-N/issues/53 (not yet addressed).
",107914493,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1302/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,