github
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
274161964 | MDU6SXNzdWUyNzQxNjE5NjQ= | 101 | TemplateAssertionError: no filter named 'tojson' | 450244 | closed | 0 | 1 | 2017-11-15T13:47:32Z | 2017-11-15T13:48:55Z | 2017-11-15T13:48:55Z | NONE | I get an exception clicking on the table link: ``` 2017-11-15 08:40:10 - (sanic)[ERROR]: Traceback (most recent call last): File "/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/sanic/app.py", line 503, in handle_request response = await response File "/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/datasette/app.py", line 155, in get return await self.view_get(request, name, hash, **kwargs) File "/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/datasette/app.py", line 219, in view_get **context, File "/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/sanic_jinja2/__init__.py", line 84, in render return html(self.render_string(template, request, **context)) File "/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/sanic_jinja2/__init__.py", line 81, in render_string return self.env.get_template(template).render(**context) File "/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py", line 812, in get_template return self._load_template(name, self.make_globals(globals)) File "/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py", line 786, in _load_template template = self.loader.load(self, name, globals) File "/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/loaders.py", line 125, in load code = environment.compile(source, name, filename) File "/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py", line 565, in compile self.handle_exception(exc_info, source_hint=source_hint) File "/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py", line 754, in handle_exception reraise(exc_type, exc_value, tb) File "/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/datasette/templates/table.html", line 29, in template <pre>params = {{ query.params|tojson(4) }}</pre> File "/Users/e/… | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/101/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
601392318 | MDU6SXNzdWU2MDEzOTIzMTg= | 101 | README should include an example of CLI data insertion | 9599 | closed | 0 | 0 | 2020-04-16T19:45:37Z | 2020-04-17T23:59:49Z | 2020-04-17T23:59:49Z | OWNER | Maybe using `curl` from the GitHub API. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/101/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274264175 | MDU6SXNzdWUyNzQyNjQxNzU= | 102 | datasette publish elasticbeanstalk | 9599 | closed | 0 | 1 | 2017-11-15T18:48:31Z | 2021-01-04T20:13:20Z | 2021-01-04T20:13:19Z | OWNER | It looks like Elastic Beanstalk is the most convenient way to deploy a docker container to AWS without first deploying a cluster. https://aws.amazon.com/blogs/devops/dockerizing-a-python-web-app/ looks helpful. We would need to automate the deployment with Boto: http://boto3.readthedocs.io/en/latest/reference/services/elasticbeanstalk.html | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/102/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
602569315 | MDU6SXNzdWU2MDI1NjkzMTU= | 102 | Can't store an array or dictionary containing a bytes value | 9599 | closed | 0 | 0 | 2020-04-18T22:49:21Z | 2020-05-01T20:45:45Z | 2020-05-01T20:45:45Z | OWNER | ``` In [1]: import sqlite_utils In [2]: db = sqlite_utils.Database(memory=True) In [3]: db["t"].insert({"id": 1, "data": {"foo": b"bytes"}}) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-3-a8ab1f72c72c> in <module> ----> 1 db["t"].insert({"id": 1, "data": {"foo": b"bytes"}}) ~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert(self, record, pk, foreign_keys, column_order, not_null, defaults, hash_id, alter, ignore, replace, extracts, conversions, columns) 950 extracts=extracts, 951 conversions=conversions, --> 952 columns=columns, 953 ) 954 ~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, conversions, columns, upsert) 1052 for key in all_columns: 1053 value = jsonify_if_needed( -> 1054 record.get(key, None if key != hash_id else _hash(record)) 1055 ) 1056 if key in extracts: ~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in jsonify_if_needed(value) 1318 def jsonify_if_needed(value): 1319 if isinstance(value, (dict, list, tuple)): -> 1320 return json.dumps(value) 1321 elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)): 1322 return value.isoformat() /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw) 229 cls is None and indent is None and separators is None and 230 default is None… | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/102/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274265878 | MDU6SXNzdWUyNzQyNjU4Nzg= | 103 | datasette publish appengine | 9599 | closed | 0 | 1 | 2017-11-15T18:54:18Z | 2021-01-04T20:05:14Z | 2021-01-04T20:05:14Z | OWNER | Similar approach to Heroku, discussed in #90 Looks like this could be pretty easy: https://cloud.google.com/appengine/docs/flexible/python/quickstart | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/103/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
610517472 | MDU6SXNzdWU2MTA1MTc0NzI= | 103 | sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns | 32605365 | closed | 0 | 8 | 2020-05-01T02:26:14Z | 2020-05-14T00:18:57Z | 2020-05-14T00:18:57Z | CONTRIBUTOR | If using insert_all to put in 1000 rows of data with varying number of columns, it comes up with this message `sqlite3.OperationalError: too many SQL variables` if the number of columns is larger in later records (past the first row) I've reduced `SQLITE_MAX_VARS` by 100 to 899 at the top of `db.py` to add wiggle room, so that if the column count increases it wont go past SQLite's batch limit as calculated by this line of code based on the count of the first row's dict keys batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns)) | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/103/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274284246 | MDExOlB1bGxSZXF1ZXN0MTUyODcwMDMw | 104 | [WIP] Add publish to heroku support | 21148 | closed | 0 | 6 | 2017-11-15T19:56:22Z | 2017-11-21T20:55:05Z | 2017-11-21T20:55:05Z | CONTRIBUTOR | simonw/datasette/pulls/104 | Refs #90 | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/104/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
610853393 | MDU6SXNzdWU2MTA4NTMzOTM= | 104 | --schema option to "sqlite-utils tables" | 9599 | closed | 0 | 0 | 2020-05-01T16:55:49Z | 2020-05-01T17:12:37Z | 2020-05-01T17:12:37Z | OWNER | Adds output showing the table schema. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/104/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274314940 | MDU6SXNzdWUyNzQzMTQ5NDA= | 105 | Consider data-package as a format for metadata | 9599 | closed | 0 | 4 | 2017-11-15T21:43:34Z | 2017-11-20T19:50:53Z | 2017-11-20T19:50:53Z | OWNER | http://frictionlessdata.io/specs/data-package/ | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/105/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
610853576 | MDU6SXNzdWU2MTA4NTM1NzY= | 105 | "sqlite-utils views" command | 9599 | closed | 0 | 1 | 2020-05-01T16:56:11Z | 2020-05-01T20:40:07Z | 2020-05-01T20:38:36Z | OWNER | Similar to `sqlite-utils tables`. See also #104. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/105/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274315193 | MDU6SXNzdWUyNzQzMTUxOTM= | 106 | Document how pagination works | 9599 | closed | 0 | 1 | 2017-11-15T21:44:32Z | 2019-06-24T06:42:33Z | 2019-06-24T06:42:33Z | OWNER | I made a start at that in this comment: https://news.ycombinator.com/item?id=15691926 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/106/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
611216862 | MDU6SXNzdWU2MTEyMTY4NjI= | 106 | create_view(..., ignore=True, replace=True) parameters | 9599 | closed | 0 | 1 | 2020-05-02T15:45:21Z | 2020-05-02T16:04:51Z | 2020-05-02T16:02:10Z | OWNER | Two new parameters which specify what should happen if the view already exists. I want this for https://github.com/dogsheep/github-to-sqlite/issues/37 Here's the current `create_view()` implementation: https://github.com/simonw/sqlite-utils/blob/b4d953d3ccef28bb81cea40ca165a647b59971fa/sqlite_utils/db.py#L325-L332 `ignore=True` will not do anything if the view exists already. `replace=True` will drop and redefine the view - but only if its SQL definition differs, otherwise it will be left alone. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/106/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274343647 | MDExOlB1bGxSZXF1ZXN0MTUyOTE0NDgw | 107 | add support for ?field__isnull=1 | 3433657 | closed | 0 | 4 | 2017-11-15T23:36:36Z | 2017-11-17T15:12:29Z | 2017-11-17T13:29:22Z | CONTRIBUTOR | simonw/datasette/pulls/107 | Is this what you had in mind for [this issue](https://github.com/simonw/datasette/issues/64)? | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/107/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
611222968 | MDU6SXNzdWU2MTEyMjI5Njg= | 107 | sqlite-utils create-view CLI command | 9599 | closed | 0 | 2 | 2020-05-02T16:15:13Z | 2020-05-03T15:36:58Z | 2020-05-03T15:36:37Z | OWNER | Can go with #27 - `sqlite-utils create-table`. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/107/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274374317 | MDU6SXNzdWUyNzQzNzQzMTc= | 108 | Include version in python code, output in template | 9599 | closed | 0 | 0 | 2017-11-16T02:32:40Z | 2017-11-16T15:30:04Z | 2017-11-16T15:30:04Z | OWNER | It would be useful if I could tell which version of datasette was running on a site. Embed version number and include it in maybe a tooltip on the “powered by datasette” link | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/108/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
611326701 | MDU6SXNzdWU2MTEzMjY3MDE= | 108 | Documentation unit tests for CLI commands | 9599 | closed | 0 | 2 | 2020-05-03T03:58:42Z | 2020-05-03T04:13:57Z | 2020-05-03T04:13:57Z | OWNER | Have a test that ensures all CLI commands are documented. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/108/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274378301 | MDU6SXNzdWUyNzQzNzgzMDE= | 109 | Set up readthedocs | 9599 | closed | 0 | 1 | 2017-11-16T02:58:01Z | 2017-11-16T16:53:26Z | 2017-11-16T16:13:56Z | OWNER | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/109/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
612658444 | MDU6SXNzdWU2MTI2NTg0NDQ= | 109 | table.create_index(..., ignore=True) | 9599 | closed | 0 | 1 | 2020-05-05T14:44:21Z | 2020-05-05T14:46:53Z | 2020-05-05T14:46:53Z | OWNER | Option to silently do nothing if the index already exists. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/109/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274578142 | MDU6SXNzdWUyNzQ1NzgxNDI= | 110 | Add --load-extension option to datasette for loading extra SQLite extensions | 9599 | closed | 0 | 2 | 2017-11-16T16:26:19Z | 2017-11-16T18:38:30Z | 2017-11-16T16:58:50Z | OWNER | This would allow users with extra SQLite extensions installed (like spatialite) to load them at runtime. Inspired by this comment: https://github.com/simonw/datasette/issues/46#issuecomment-344810525 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/110/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
613755043 | MDU6SXNzdWU2MTM3NTUwNDM= | 110 | Support decimal.Decimal type | 134771 | closed | 0 | 6 | 2020-05-07T03:57:19Z | 2020-05-11T01:58:20Z | 2020-05-11T01:50:11Z | NONE | Decimal types in Postgres cause a failure in db.py data type selection --- I have a Django app using a MoneyField, which uses a `numeric(14,0)` data type in Postgres (https://www.postgresql.org/docs/9.3/datatype-numeric.html). When attempting to export that table I get the following error: ```bash $ db-to-sqlite --table isaweb_proposal "postgres://connection" test.db .... column_type=COLUMN_TYPE_MAPPING[column_type], KeyError: <class 'decimal.Decimal'> ``` Looking at `sql_utils.db.py` at 292-ish it's clear that there is no matching type for what I assume SQLAlchemy interprets as Python decimal.Decimal. From the [SQLite docs](https://www.sqlite.org/datatype3.html#affinity_name_examples) it looks like DECIMAL in other DBs are considered numeric. I'm not quite sure if it's as simple as adding a data type to that list or if there are repercussions beyond it. Thanks for a great tool! | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/110/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274615452 | MDU6SXNzdWUyNzQ2MTU0NTI= | 111 | Add “updated” to metadata | 9599 | open | 0 | 12 | 2017-11-16T18:22:20Z | 2021-09-21T22:48:27Z | OWNER | To give an indication as to when the data was last updated. This should be a field in the metadata that is then shown on the index page and in the footer, if it is set. Also support setting it using an option to “datasette publish” and “datasette package” - which can either be a string or can be the magic string “today” to set it to today’s date: datasette publish file.db --updated=today | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/111/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
615477131 | MDU6SXNzdWU2MTU0NzcxMzE= | 111 | sqlite-utils drop-table and drop-view commands | 9599 | closed | 0 | 2 | 2020-05-10T21:10:42Z | 2020-05-11T01:58:36Z | 2020-05-11T00:44:26Z | OWNER | Would be useful to be able to drop views and tables from the CLI. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/111/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274617240 | MDU6SXNzdWUyNzQ2MTcyNDA= | 112 | Allow --load-extension to be set via environment variables | 9599 | closed | 0 | 1 | 2017-11-16T18:28:31Z | 2017-11-17T14:19:23Z | 2017-11-17T14:17:27Z | OWNER | This will make it easier to package up datasette in a Docker container with a bunch of pre-compiled extensions without the user having to remember to include all of the options every time. Click has a mechanism for this: http://click.pocoo.org/5/options/#multiple-values-from-environment-values | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/112/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
616271236 | MDU6SXNzdWU2MTYyNzEyMzY= | 112 | add_foreign_key(...., ignore=True) | 9599 | closed | 0 | 5896742 | 4 | 2020-05-12T00:24:00Z | 2020-09-20T22:17:34Z | 2020-09-20T22:17:34Z | OWNER | When using this library I often find myself wanting to "add this foreign key, but only if it doesn't exist yet". The `ignore=True` parameter is increasingly being used for this else where in the library (e.g. in `create_view()`). | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/112/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
274662378 | MDU6SXNzdWUyNzQ2NjIzNzg= | 113 | Fix the bug on the database custom SQL query view | 9599 | closed | 0 | 2919870 | 0 | 2017-11-16T21:01:26Z | 2017-11-17T15:40:52Z | 2017-11-17T15:40:52Z | OWNER | https://sf-film-locations.now.sh/sf-film-locations-57704b7?sql=select+*+from+Film_Locations_in_San_Francisco <img width="852" alt="sf-film-locations" src="https://user-images.githubusercontent.com/9599/32915540-19bad984-cace-11e7-9ed3-bd7294891dc0.png"> This is the bug I fixed in 01e0c3fa18cd0dd7970e208790ffd683a420c924 - but I only fixed it in one place. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/113/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
621286870 | MDU6SXNzdWU2MjEyODY4NzA= | 113 | Syntactic sugar for ATTACH DATABASE | 9599 | closed | 0 | 2 | 2020-05-19T21:10:00Z | 2021-02-19T05:09:12Z | 2021-02-19T04:56:36Z | OWNER | https://www.sqlite.org/lang_attach.html Maybe something like this: ```python db.attach("other_db", "other_db.db") ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/113/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274733145 | MDExOlB1bGxSZXF1ZXN0MTUzMjAxOTQ1 | 114 | Add spatialite, switch to debian and local build | 54999 | closed | 0 | 1 | 2017-11-17T02:37:09Z | 2017-11-17T03:50:52Z | 2017-11-17T03:50:52Z | CONTRIBUTOR | simonw/datasette/pulls/114 | Improves the Dockerfile to support spatial datasets, work with the local datasette code (Friendly with git tags and Dockerhub) and moves to slim debian, a small image easy to extend via apt packages for sqlite. | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/114/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
621989740 | MDU6SXNzdWU2MjE5ODk3NDA= | 114 | table.transform() method for advanced alter table | 9599 | closed | 0 | 5897911 | 26 | 2020-05-20T18:20:46Z | 2020-09-22T07:51:37Z | 2020-09-22T04:20:02Z | OWNER | SQLite's `ALTER TABLE` can only do the following: * Rename a table * Rename a column * Add a column Notably, it cannot drop columns - so tricks like "add a float version of this text column, populate it, then drop the old one and rename" won't work. The docs here https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes describe a way of implementing full alters safely within a transaction, but it's fiddly. 1. Create new table 2. Copy data 3. Drop old table 4. Rename new into old It would be great if `sqlite-utils` provided an abstraction to help make these kinds of changes safely. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
274877366 | MDExOlB1bGxSZXF1ZXN0MTUzMzA2ODgy | 115 | Add keyboard shortcut to execute SQL query | 198537 | closed | 0 | 1 | 2017-11-17T14:13:33Z | 2017-11-17T15:16:34Z | 2017-11-17T14:22:56Z | CONTRIBUTOR | simonw/datasette/pulls/115 | Very cool tool, thanks a lot! This PR adds a `Shift-Enter` short cut to execute the SQL query. I used CodeMirrors keyboard handling. | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/115/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
637889964 | MDU6SXNzdWU2Mzc4ODk5NjQ= | 115 | Ability to execute insert/update statements with the CLI | 9599 | closed | 0 | 1 | 2020-06-12T17:01:17Z | 2020-06-12T17:51:11Z | 2020-06-12T17:41:10Z | OWNER | ``` $ sqlite-utils github.db "update stars set starred_at = ''" Traceback (most recent call last): File "/Users/simon/.local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/cli.py", line 673, in query headers = [c[0] for c in cursor.description] TypeError: 'NoneType' object is not iterable ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/115/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274884209 | MDU6SXNzdWUyNzQ4ODQyMDk= | 116 | Add documentation section about SQLite extensions | 9599 | closed | 0 | 1 | 2017-11-17T14:36:30Z | 2018-05-28T17:23:42Z | 2018-05-28T17:23:41Z | OWNER | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/116/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
644122661 | MDU6SXNzdWU2NDQxMjI2NjE= | 116 | Documentation for table.pks introspection property | 9599 | closed | 0 | 2 | 2020-06-23T20:27:24Z | 2020-06-23T21:21:33Z | 2020-06-23T21:03:14Z | OWNER | https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L535-L540 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/116/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
274900388 | MDExOlB1bGxSZXF1ZXN0MTUzMzI0MzAx | 117 | Don't prevent tabbing to `Run SQL` button | 198537 | closed | 0 | 1 | 2017-11-17T15:27:50Z | 2017-11-19T20:30:24Z | 2017-11-18T00:53:43Z | CONTRIBUTOR | simonw/datasette/pulls/117 | Mentioned in #115 Here you go! | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/117/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
644161221 | MDU6SXNzdWU2NDQxNjEyMjE= | 117 | Support for compound (composite) foreign keys | 9599 | open | 0 | 3 | 2020-06-23T21:33:42Z | 2020-06-23T21:40:31Z | OWNER | It turns out SQLite supports composite foreign keys: https://www.sqlite.org/foreignkeys.html#fk_composite Their example looks like this: ```sql CREATE TABLE album( albumartist TEXT, albumname TEXT, albumcover BINARY, PRIMARY KEY(albumartist, albumname) ); CREATE TABLE song( songid INTEGER, songartist TEXT, songalbum TEXT, songname TEXT, FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) ); ``` Here's what that looks like in sqlite-utils: ``` In [1]: import sqlite_utils In [2]: import sqlite3 In [3]: conn = sqlite3.connect(":memory:") In [4]: conn Out[4]: <sqlite3.Connection at 0x1087186c0> In [5]: conn.executescript(""" ...: CREATE TABLE album( ...: albumartist TEXT, ...: albumname TEXT, ...: albumcover BINARY, ...: PRIMARY KEY(albumartist, albumname) ...: ); ...: ...: CREATE TABLE song( ...: songid INTEGER, ...: songartist TEXT, ...: songalbum TEXT, ...: songname TEXT, ...: FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) ...: ); ...: """) Out[5]: <sqlite3.Cursor at 0x1088def10> In [6]: db = sqlite_utils.Database(conn) In [7]: db.tables … | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/117/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
275048699 | MDExOlB1bGxSZXF1ZXN0MTUzNDMyMDQ1 | 118 | Foreign key information on row and table pages | 9599 | closed | 0 | 0 | 2017-11-18T03:13:27Z | 2017-11-18T03:15:57Z | 2017-11-18T03:15:50Z | OWNER | simonw/datasette/pulls/118 | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/118/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
651844316 | MDExOlB1bGxSZXF1ZXN0NDQ1MDIzMzI2 | 118 | Add insert --truncate option | 79913 | closed | 0 | 9 | 2020-07-06T21:58:40Z | 2020-07-08T17:26:21Z | 2020-07-08T17:26:21Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/118 | Deletes all rows in the table (if it exists) before inserting new rows. SQLite doesn't implement a TRUNCATE TABLE statement but does optimize an unqualified DELETE FROM. This can be handy if you want to refresh the entire contents of a table but a) don't have a PK (so can't use --replace), b) don't want the table to disappear (even briefly) for other connections, and c) have to handle records that used to exist being deleted. Ideally the replacement of rows would appear instantaneous to other connections by putting the DELETE + INSERT in a transaction, but this is very difficult without breaking other code as the current transaction handling is inconsistent and non-systematic. There exists the possibility for the DELETE to succeed but the INSERT to fail, leaving an empty table. This is not much worse, however, than the current possibility of one chunked INSERT succeeding and being committed while the next chunked INSERT fails, leaving a partially complete operation. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
275082158 | MDU6SXNzdWUyNzUwODIxNTg= | 119 | Build an "export this data to google sheets" plugin | 9599 | closed | 0 | 1 | 2017-11-18T14:14:51Z | 2020-06-04T18:46:40Z | 2020-06-04T18:46:39Z | OWNER | Inspired by https://github.com/kren1/tosheets It should be a plug-in because I'd like to keep all interactions with proprietary / non-open-source software encapsulated in plugins rather than shipped as part of core. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/119/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
652700770 | MDU6SXNzdWU2NTI3MDA3NzA= | 119 | Ability to remove a foreign key | 9599 | closed | 0 | 3 | 2020-07-07T22:31:37Z | 2020-09-24T20:36:59Z | 2020-09-24T20:36:59Z | OWNER | Useful if you add one but make a mistake and need to undo it without recreating the database from scratch. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/119/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275087397 | MDU6SXNzdWUyNzUwODczOTc= | 120 | Plugin that adds an authentication layer of some sort | 9599 | closed | 0 | 4 | 2017-11-18T15:39:13Z | 2020-03-16T18:48:06Z | 2020-03-16T18:48:06Z | OWNER | Would allow people who want to host private data to do so. .sh | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/120/reactions", "total_count": 7, "+1": 5, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 2, "rocket": 0, "eyes": 0 } |
completed | ||||||
652816158 | MDExOlB1bGxSZXF1ZXN0NDQ1ODMzOTA4 | 120 | Fix query command's support for DML | 79913 | closed | 0 | 1 | 2020-07-08T01:36:34Z | 2020-07-08T05:14:04Z | 2020-07-08T05:14:04Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/120 | See commit messages for details. I ran into this while investigating another feature/issue. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/120/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
275089535 | MDU6SXNzdWUyNzUwODk1MzU= | 121 | ?_json=foo&_json=bar query string argument | 9599 | closed | 0 | 4 | 2017-11-18T16:09:55Z | 2018-05-31T13:48:12Z | 2018-05-28T18:11:51Z | OWNER | Causes the specified columns in the output to be treated as JSON, and returned deserialized in the .json or .jsono response. This will be particularly powerful when combined with https://sqlite.org/json1.html | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/121/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
652961907 | MDU6SXNzdWU2NTI5NjE5MDc= | 121 | Improved (and better documented) support for transactions | 9599 | open | 0 | 3 | 2020-07-08T04:56:51Z | 2020-09-24T20:36:46Z | OWNER | _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393_ We should put some thought into how this library supports and encourages smart use of transactions. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/121/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
275092453 | MDU6SXNzdWUyNzUwOTI0NTM= | 122 | Redesign JSON output, ditch jsono, offer variants controlled by parameter instead | 9599 | closed | 0 | 5 | 2017-11-18T16:52:28Z | 2018-04-08T14:54:09Z | 2018-04-08T14:54:09Z | OWNER | I want to support three variants for the rows output: * a list of lists, with a columns key saying what they are * a list of dictionaries * a single dictionary where the keys are the primary keys of the rows and the values are the row dictionaries themselves I also want to make the various bits of metadata opt-in - so you don't get the SQL statement unless you ask for it. These output options should be controlled by query string arguments. I will set the .jsono URL to redirect to .json with the corresponding options. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/122/reactions", "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
665700495 | MDU6SXNzdWU2NjU3MDA0OTU= | 122 | CLI utility for inserting binary files into SQLite | 9599 | closed | 0 | 10 | 2020-07-26T03:27:39Z | 2020-07-27T07:10:41Z | 2020-07-27T07:09:03Z | OWNER | SQLite BLOB columns can store entire binary files. The challenge is inserting them, since they don't neatly fit into JSON objects. It would be great if the `sqlite-utils` CLI had a trick for helping with this. Inspired by https://github.com/simonw/datasette-media/issues/14 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/122/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275125561 | MDU6SXNzdWUyNzUxMjU1NjE= | 123 | Datasette serve should accept paths/URLs to CSVs and other file formats | 9599 | open | 0 | 9 | 2017-11-19T02:05:48Z | 2021-07-19T00:04:32Z | OWNER | This would remove the csvs-to-sqlite step which I end up using for almost everything. I'm hesitant to introduce pandas as a required dependency though since it require compiling numpy. Could build it so this option is only available if you have pandas installed. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/123/reactions", "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0 } |
||||||||
665701216 | MDU6SXNzdWU2NjU3MDEyMTY= | 123 | --raw option for outputting binary content | 9599 | closed | 0 | 0 | 2020-07-26T03:35:39Z | 2020-07-26T16:44:11Z | 2020-07-26T16:44:11Z | OWNER | Related to the `insert-files` work in #122 - it should be easy to get binary data back out of the database again. One way to do that could be: sqlite-utils files.db "select content from files where key = 'foo.jpg'" --raw The `--raw` option would cause just the contents of the first column to be output directly to stdout. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/123/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275125805 | MDU6SXNzdWUyNzUxMjU4MDU= | 124 | Option to open readonly but not immutable | 9599 | closed | 0 | 5 | 2017-11-19T02:11:03Z | 2019-06-24T06:43:46Z | 2019-06-24T06:43:46Z | OWNER | Immutable assumes no other process can modify the file. An option to open reqdonly instead would enable other processes to update the file in place. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/124/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
665802405 | MDU6SXNzdWU2NjU4MDI0MDU= | 124 | sqlite-utils query should support named parameters | 9599 | closed | 0 | 1 | 2020-07-26T15:25:10Z | 2020-07-30T22:57:51Z | 2020-07-27T03:53:58Z | OWNER | To help out with escaping - so you can run this: sqlite-utils query "insert into foo (blah) values (:blah)" --param blah `something here` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/124/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275135393 | MDU6SXNzdWUyNzUxMzUzOTM= | 125 | Plot rows on a map with Leaflet and Leaflet.markercluster | 9599 | closed | 0 | 2 | 2017-11-19T06:05:05Z | 2018-04-26T15:14:31Z | 2018-04-26T15:14:31Z | OWNER | https://github.com/Leaflet/Leaflet.markercluster would allow us to paginate-load in an enormous set of rows with latitude/longitude points, e.g. https://australian-dunnies.now.sh/ Here's a demo of it loading 50,000 markers: https://leaflet.github.io/Leaflet.markercluster/example/marker-clustering-realworld.50000.html - and it looks like it's easy to support progress bars for if we were iteratively loading 1,000 markers at a time using datasette pagination. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/125/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
665817570 | MDU6SXNzdWU2NjU4MTc1NzA= | 125 | Output binary columns in "sqlite-utils query" JSON | 9599 | closed | 0 | 4 | 2020-07-26T16:47:02Z | 2020-07-27T00:49:41Z | 2020-07-27T00:48:45Z | OWNER | You get an error if you try to run a query that returns data from a BLOB. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/125/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275135535 | MDU6SXNzdWUyNzUxMzU1MzU= | 126 | Blog entry announcing foreign key support | 9599 | closed | 0 | 2919870 | 1 | 2017-11-19T06:09:06Z | 2017-11-30T16:49:24Z | 2017-11-30T16:49:24Z | OWNER | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/126/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
665819048 | MDU6SXNzdWU2NjU4MTkwNDg= | 126 | Ability to insert binary data on the CLI using JSON | 9599 | closed | 0 | 2 | 2020-07-26T16:54:14Z | 2020-07-27T04:00:33Z | 2020-07-27T03:59:45Z | OWNER | > I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says "these columns are base64 encoded, store the decoded data in a BLOB". > > That would solve inserting binary data using JSON too. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247_ | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/126/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275135719 | MDU6SXNzdWUyNzUxMzU3MTk= | 127 | Filtered tables should show count of all matching rows, if fast enough | 9599 | closed | 0 | 2919870 | 2 | 2017-11-19T06:13:29Z | 2017-11-24T22:02:01Z | 2017-11-24T22:02:01Z | OWNER | Relates to #86. If you are viewing a filtered page e.g. https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9/bob-ross%2Felements-by-episode?CLOUDS=1 we should show the count of matching rows. Since this could be an expensive operation, we will run it with a strict time limit (maybe 50ms). If the time limit is exceeded we will display "many" instead, perhaps? Maybe even link to a count(*) query that would get the full 1000ms time limit which the user can click on if they like (that could even Ajax-in the result). | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/127/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
666040390 | MDU6SXNzdWU2NjYwNDAzOTA= | 127 | Ability to insert files piped to insert-files stdin | 9599 | closed | 0 | 3 | 2020-07-27T07:09:33Z | 2020-07-30T03:08:52Z | 2020-07-30T03:08:18Z | OWNER | > Inserting files by piping them in should work - but since a filename cannot be derived this will need a `--name blah.gif` option. > > cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif > _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071_ | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/127/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275159710 | MDU6SXNzdWUyNzUxNTk3MTA= | 128 | Every visualization should have an "embed" button | 9599 | open | 0 | 0 | 2017-11-19T13:38:13Z | 2019-05-13T18:33:51Z | OWNER | At least for the first round of visualizations, any time you construct one using the UI the result should include an "embed this" button that returns source code to copy and paste These examples should use unpkg.com (or similarl) urls with SRI hashes, eg https://www.srihash.org - and should load data from the datasette JSON API. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/128/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
666639051 | MDU6SXNzdWU2NjY2MzkwNTE= | 128 | Support UUID and memoryview types | 9599 | closed | 0 | 1 | 2020-07-27T23:08:34Z | 2020-07-30T01:10:43Z | 2020-07-30T01:10:43Z | OWNER | `psycopg2` can return data from PostgreSQL as `uuid.UUID` or `memoryview` objects. These should to be supported by `sqlite-utils` - mainly for https://github.com/simonw/db-to-sqlite | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/128/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275164558 | MDU6SXNzdWUyNzUxNjQ1NTg= | 129 | Hide FTS-created tables by default on the database index page | 9599 | closed | 0 | 2 | 2017-11-19T14:50:42Z | 2017-11-22T20:22:02Z | 2017-11-22T20:19:04Z | OWNER | SQLite databases that use FTS include a number of automatically generated tables, e.g.: https://sf-trees-search.now.sh/sf-trees-search-a899b92 <img width="730" alt="sf-trees-search_and_sf-trees-search" src="https://user-images.githubusercontent.com/9599/32991960-bf05abee-ccf9-11e7-8bd1-54dcde4ca491.png"> Of these, only the `Street_Tree_List` table is actually relevant to the user. We can detect which tables are FTS tables by first finding the virtual tables: sqlite> .headers on sqlite> select * from sqlite_master where rootpage = 0; type|name|tbl_name|rootpage|sql table|Search|Search|0|CREATE VIRTUAL TABLE "Street_Tree_List_fts" USING FTS4 ("qAddress", "qCaretaker", "qSpecies") Then parsing the above to figure out which ones are USING FTS? - then assume that any table which starts with that `Street_Tree_List_fts` prefix was created to support search: sqlite> select * from sqlite_master where type='table' and tbl_name like 'Street_Tree_List_fts%'; type|name|tbl_name|rootpage|sql table|Search_content|Search_content|10355|CREATE TABLE 'Street_Tree_List_fts_content'(docid INTEGER PRIMARY KEY, 'c0qAddress', 'c1qCaretaker', 'c2qSpecies') table|Search_segments|Search_segments|10356|CREATE TABLE 'Street_Tree_List_fts_segments'(blockid INTEGER PRIMARY KEY, block BLOB) table|Search_segdir|Search_segdir|10357|CREATE TABLE 'Street_Tree_List_fts_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx)) table|Search_docsize|Search_docsize|10359|CREATE TABLE 'Street_Tree_List_fts_docsize'(docid INTEGER PRIMARY KEY, size BLOB) table|Search_stat|Search_stat|10360|CREATE TABLE 'Street_Tree_List_fts_stat'(id INTEGER PRIMARY KEY, value BLOB) We won't hide these completely - instead, we'll default the database index view to not showing them with a message that says "5 hidden tables" and support ?_hidden=1 to display them. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/129/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
668308777 | MDU6SXNzdWU2NjgzMDg3Nzc= | 129 | "insert-files --sqlar" for creating SQLite archives | 9599 | closed | 0 | 2 | 2020-07-30T02:28:29Z | 2020-07-30T22:41:01Z | 2020-07-30T22:40:55Z | OWNER | A `--sqlar` option could cause `insert-files` to behave in the same way as SQLite's own sqlar mechanism. https://www.sqlite.org/sqlar.html and https://sqlite.org/sqlar/doc/trunk/README.md | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/129/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275166078 | MDU6SXNzdWUyNzUxNjYwNzg= | 130 | Rename "datasette build" to "datasette inspect" | 9599 | closed | 0 | 0 | 2017-11-19T15:08:02Z | 2017-12-07T16:57:58Z | 2017-12-07T16:57:58Z | OWNER | This command introspects the databases and writes out a JSON summary. I think I'd like to use `datasette build` for something more interesting, potentially duplicating functionality from https://github.com/simonw/csvs-to-sqlite Since the internal method that does this is called `ds.inspect()` that seems like a reasonable replacement name for the command. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/130/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
671130371 | MDU6SXNzdWU2NzExMzAzNzE= | 130 | Support tokenize option for FTS | 9599 | closed | 0 | 3 | 2020-08-01T19:27:22Z | 2020-08-01T20:51:28Z | 2020-08-01T20:51:14Z | OWNER | FTS5 supports things like porter stemming using a `tokenize=` option: https://www.sqlite.org/fts5.html#tokenizers Something like this in code: ``` CREATE VIRTUAL TABLE [{table}_fts] USING {fts_version} ( {columns}, tokenize='porter', content=[{table}] ); ``` I tried this out just now and it worked exactly as expected. So... `db[table].enable_fts(...) should accept a 'tokenize=` argument, and `sqlite-utils enable-fts ...` should support a `--tokenize` option. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/130/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275166669 | MDU6SXNzdWUyNzUxNjY2Njk= | 131 | UI support for running FTS searches | 9599 | closed | 0 | 3 | 2017-11-19T15:16:20Z | 2017-11-19T17:18:05Z | 2017-11-19T17:00:12Z | OWNER | Here's an example query that searches all FTS indexed columns in a table: https://sf-trees-search.now.sh/sf-trees-search-a899b92?sql=select+*+from+Street_Tree_List+where+rowid+in+%28select+rowid+from+Street_Tree_List_fts+where+Street_Tree_List_fts+match+%27grove+london+dpw%27%29%0D%0A And here's a query that searches a specific column: https://sf-trees-search.now.sh/sf-trees-search-a899b92?sql=select+*+from+Street_Tree_List+where+rowid+in+%28select+rowid+from+Street_Tree_List_fts+where+qSpecies+match+%27london%27%29%0D%0A If we detect that a table has FTS enabled (which we can do by looking for it as a content table reference in another FTS table's create definition) we should add a search box to the table page which constructs this query - maybe using `?_search=XXX` in the query string? <s>To support search against specified columns, we can do `?_search__ qSpecies=London`.</s> - not necessary, see comment below. - [x] Detect if a table has a FTS index defined against it as a content= parameter - [x] Decide what to do if there is more than one FTS index (maybe just pick the first one?) - [x] Add the `?_search=` query string argument - [x] Add the UI | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/131/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
675753042 | MDU6SXNzdWU2NzU3NTMwNDI= | 131 | sqlite-utils insert: options for column types | 9599 | open | 0 | 5 | 2020-08-09T18:59:11Z | 2022-03-15T13:21:42Z | OWNER | The `insert` command currently results in string types for every column - at least when used against CSV or TSV inputs. It would be useful if you could do the following: - automatically detects the column types based on eg the first 1000 records - explicitly state the rule for specific columns `--detect-types` could work for the former - or it could do that by default and allow opt-out using `--no-detect-types` For specific columns maybe this: sqlite-utils insert db.db images images.tsv \ --tsv \ -c id int \ -c score float | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/131/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
275175929 | MDU6SXNzdWUyNzUxNzU5Mjk= | 132 | Row view is not currently expanding foreign keys | 9599 | closed | 0 | 2919870 | 1 | 2017-11-19T17:24:25Z | 2017-11-23T21:51:51Z | 2017-11-23T21:51:30Z | OWNER | Eg https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List/1 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/132/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
675839512 | MDU6SXNzdWU2NzU4Mzk1MTI= | 132 | Features for enabling and disabling WAL mode | 9599 | closed | 0 | 5 | 2020-08-10T03:25:44Z | 2020-08-10T18:59:35Z | 2020-08-10T18:59:35Z | OWNER | I finally figured out how to enable WAL - turns out it's a property of the database file itself: https://github.com/simonw/til/blob/master/sqlite/enabling-wal-mode.md | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/132/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275176006 | MDU6SXNzdWUyNzUxNzYwMDY= | 133 | If view is filtered, search should apply within those filtered rows | 9599 | closed | 0 | 2919870 | 3 | 2017-11-19T17:25:36Z | 2017-11-24T22:30:32Z | 2017-11-24T22:30:15Z | OWNER | Eg on https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List?qSpecies=1 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/133/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
677839979 | MDU6SXNzdWU2Nzc4Mzk5Nzk= | 133 | Release a sdist to PyPI | 9599 | closed | 0 | 1 | 2020-08-12T16:55:09Z | 2020-08-12T17:05:06Z | 2020-08-12T17:05:06Z | OWNER | https://pypi.org/project/sqlite-utils/#files currently just has a wheel. I need this to package for homebrew: https://github.com/simonw/homebrew-datasette/issues/10 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/133/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275176094 | MDU6SXNzdWUyNzUxNzYwOTQ= | 134 | Filtered table view should show a count | 9599 | closed | 0 | 2919870 | 1 | 2017-11-19T17:26:53Z | 2017-11-19T18:10:49Z | 2017-11-19T18:10:49Z | OWNER | Let's do the thing where we attempt to show an accurate count if it can be done in less than 50ms | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/134/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
683804172 | MDU6SXNzdWU2ODM4MDQxNzI= | 134 | --load-extension option for sqlite-utils query | 9599 | closed | 0 | 4 | 2020-08-21T20:12:42Z | 2020-08-21T21:06:26Z | 2020-08-21T20:54:19Z | OWNER | I got this error: ``` % sqlite-utils calands.db 'create table superunits_with_maps_view_concrete as select * from superunits_with_maps_view' Traceback (most recent call last): ... cursor = db.conn.execute(sql, dict(param)) sqlite3.OperationalError: no such function: AsGeoJSON ``` A `--load-extension=/usr/local/lib/mod_spatialite.dylib` option (imitating the same option for Datasette) would help. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/134/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275179724 | MDU6SXNzdWUyNzUxNzk3MjQ= | 135 | ?_search=x should work if used directly against a FTS virtual table | 9599 | closed | 0 | 2949431 | 3 | 2017-11-19T18:17:53Z | 2017-12-07T04:54:41Z | 2017-12-07T04:54:41Z | OWNER | e.g. https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List_fts?_search=grove should work | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/135/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
683805434 | MDU6SXNzdWU2ODM4MDU0MzQ= | 135 | Code for finding SpatiaLite in the usual locations | 9599 | closed | 0 | 3 | 2020-08-21T20:15:34Z | 2022-02-05T00:04:26Z | 2020-08-21T20:30:13Z | OWNER | I built this for `shapefile-to-sqlite` but it would be useful in `sqlite-utils` too: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L16-L19 ```python SPATIALITE_PATHS = ( "/usr/lib/x86_64-linux-gnu/mod_spatialite.so", "/usr/local/lib/mod_spatialite.dylib", ) ``` https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L105-L109 ```python def find_spatialite(): for path in SPATIALITE_PATHS: if os.path.exists(path): return path return None ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/135/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275228834 | MDU6SXNzdWUyNzUyMjg4MzQ= | 136 | "Reformat SQL" button next to SQL editor textarea | 9599 | closed | 0 | 0 | 2017-11-20T03:42:19Z | 2019-10-14T03:46:13Z | 2019-10-14T03:46:13Z | OWNER | Can use this: https://github.com/zeroturnaround/sql-formatter https://zeroturnaround.github.io/sql-formatter/ | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/136/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
683812642 | MDU6SXNzdWU2ODM4MTI2NDI= | 136 | --load-extension=spatialite shortcut option | 9599 | closed | 0 | 3 | 2020-08-21T20:31:25Z | 2022-02-05T00:04:26Z | 2020-10-16T19:14:32Z | OWNER | In conjunction with #135 - this would do the same thing as `--load-extension=path-to-spatialite` (see #134) | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/136/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275415799 | MDU6SXNzdWUyNzU0MTU3OTk= | 137 | Ability to combine multiple SQL queries on a single graph | 9599 | open | 0 | 1 | 2017-11-20T16:26:57Z | 2019-05-13T18:33:51Z | OWNER | This would make visualizations significantly more powerful. The interesting challenge will be around the URL design. It would be useful to be able to combine either multiple explicit SQL queries or multiple queries based on the filter string parameters passed to one or more table views. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/137/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
683830416 | MDU6SXNzdWU2ODM4MzA0MTY= | 137 | --load-extension for other sqlite-utils commands | 9599 | closed | 0 | 1 | 2020-08-21T21:12:56Z | 2020-10-16T19:14:32Z | 2020-10-16T19:14:32Z | OWNER | e.g. for this: ``` calands-datasette % sqlite-utils tables calands.db --counts [{"table": "spatial_ref_sys", "count": 4924}, {"table": "spatialite_history", "count": 14}, {"table": "sqlite_sequence", "count": 1}, {"table": "geometry_columns", "count": 2}, {"table": "spatial_ref_sys_aux", "count": 4873}, {"table": "views_geometry_columns", "count": 0}, {"table": "virts_geometry_columns", "count": 0}, {"table": "geometry_columns_statistics", "count": 2}, {"table": "views_geometry_columns_statistics", "count": 0}, {"table": "virts_geometry_columns_statistics", "count": 0}, {"table": "geometry_columns_field_infos", "count": 0}, {"table": "views_geometry_columns_field_infos", "count": 0}, {"table": "virts_geometry_columns_field_infos", "count": 0}, {"table": "geometry_columns_time", "count": 2}, {"table": "geometry_columns_auth", "count": 2}, {"table": "views_geometry_columns_auth", "count": 0}, {"table": "virts_geometry_columns_auth", "count": 0}, Traceback (most recent call last): File "/usr/local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py", line 143, in tables … | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/137/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275476839 | MDU6SXNzdWUyNzU0NzY4Mzk= | 138 | Per-database and per-table metadata, probably using data-package | 9599 | closed | 0 | 1 | 2017-11-20T19:50:10Z | 2017-12-10T03:08:36Z | 2017-12-10T03:08:26Z | OWNER | Ability to annotate databases and tables with extra metadata describing their purpose, providing source and licensing information and describing individual columns. http://frictionlessdata.io/specs/data-package/ looks like a great format for this, see #105 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/138/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
684118950 | MDU6SXNzdWU2ODQxMTg5NTA= | 138 | extracts= doesn't configure foreign keys | 9599 | closed | 0 | 2 | 2020-08-23T05:21:15Z | 2020-09-24T22:47:01Z | 2020-09-24T22:46:52Z | OWNER | In using `extracts=` for `shapefiles-to-sqlite` in https://github.com/simonw/shapefile-to-sqlite/issues/9 I've run into a couple of pretty serious flaws: - The columns in the original table are still `TEXT` even when the foreign key they are supposed to reference is an `INTEGER` - which means Datasette foreign key features don't actually work - Those foreign key relationships aren't setup automatically - creating them is left as an exercise for the developer | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/138/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275493851 | MDU6SXNzdWUyNzU0OTM4NTE= | 139 | Build a visualization plugin for Vega | 9599 | closed | 0 | 2 | 2017-11-20T20:47:41Z | 2018-07-10T17:48:18Z | 2018-07-10T17:48:18Z | OWNER | https://vega.github.io/vega/examples/population-pyramid/ for example looks pretty easy to hook up to Datasette. Depends on #14 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/139/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
686978131 | MDU6SXNzdWU2ODY5NzgxMzE= | 139 | insert_all(..., alter=True) should work for new columns introduced after the first 100 records | 96218 | closed | 0 | 7 | 2020-08-27T06:25:25Z | 2020-08-28T22:48:51Z | 2020-08-28T22:30:14Z | CONTRIBUTOR | Is there a way to make `.insert_all()` work properly when new columns are introduced outside the first 100 records (with or without the `alter=True` argument)? I'm using `.insert_all()` to bulk insert ~3-4k records at a time and it is common for records to need to introduce new columns. However, if new columns are introduced after the first 100 records, `sqlite_utils` doesn't even raise the `OperationalError: table ... has no column named ...` exception; it just silently drops the extra data and moves on. It took me a while to find this little snippet in the [documentation for `.insert_all()`](https://sqlite-utils.readthedocs.io/en/stable/python-api.html#bulk-inserts) (it's not mentioned under [Adding columns automatically on insert/update](https://sqlite-utils.readthedocs.io/en/stable/python-api.html#bulk-inserts)): > The column types used in the CREATE TABLE statement are automatically derived from the types of data in that first batch of rows. **_Any additional or missing columns in subsequent batches will be ignored._** I tried changing the `batch_size` argument to the total number of records, but it seems only to effect the number of rows that are committed at a time, and has no influence on this problem. Is there a way around this that you would suggest? It seems like it should raise an exception at least. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
275755475 | MDU6SXNzdWUyNzU3NTU0NzU= | 140 | Heatmap visualization plugin | 9599 | open | 0 | 2 | 2017-11-21T15:34:23Z | 2019-05-13T18:33:51Z | OWNER | Could use https://github.com/scottbedard/svelte-heatmap | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/140/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
688351054 | MDU6SXNzdWU2ODgzNTEwNTQ= | 140 | Idea: insert-files mechanism for adding extra columns with fixed values | 9599 | open | 0 | 1 | 2020-08-28T20:57:36Z | 2022-03-20T19:45:45Z | OWNER | Say for example you want to populate a `file_type` column with the value `gif`. That could work like this: ``` sqlite-utils insert-files gifs.db images *.gif \ -c path -c md5 -c last_modified:mtime \ -c file_type:text:gif --pk=path ``` So a column defined as a `text` column with a value that follows a second colon. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/140/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
275814941 | MDU6SXNzdWUyNzU4MTQ5NDE= | 141 | datasette publish can fail if /tmp is on a different device | 21148 | closed | 0 | 2949431 | 5 | 2017-11-21T18:28:05Z | 2020-04-29T03:27:54Z | 2017-12-08T16:06:36Z | CONTRIBUTOR | `datasette publish` uses hard links to avoid copying the db into a tmp directory. This can fail if `/tmp` is on another device, because hardlinks can't cross devices. You'll see something like this: ``` $ datasette publish heroku whatever.db ... OSError: [Errno 18] Invalid cross-device link: '/mnt/c/Users/jacob/c/datasette/whatever.db' -> '/tmp/tmpvxq2yof6/whatever.db' ``` [In my case this is failing because I'm on a Windows machine, using WSL, so my code's on a different virtual filesystem from the Linux subsystem, Because Reasons.] I'm not sure if it's possible to detect this (can you figure out which device `/tmp` is on?), or what the fallback should be (soft link? copy?). | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/141/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
688352145 | MDU6SXNzdWU2ODgzNTIxNDU= | 141 | insert-files support for compressed values | 9599 | open | 0 | 0 | 2020-08-28T20:59:46Z | 2020-09-24T20:36:08Z | OWNER | The `sqlar` format supports this, it would be useful if `insert-files` could support this too. https://www.sqlite.org/sqlar.html | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/141/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
275917760 | MDU6SXNzdWUyNzU5MTc3NjA= | 142 | Show extra instructions with the interrupted | 9599 | closed | 0 | 3 | 2017-11-22T01:44:29Z | 2018-05-28T21:25:06Z | 2018-05-28T21:24:35Z | OWNER | When you are using Datasette locally for ad-hoc analysis it can be frustrating to hit the time limit. If you start it with the correct command line arguments you can disable that time limit. So how about we tell you how to do that anytime you hit the interrupted error provided you are accessing it from localhost. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/142/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
688386219 | MDExOlB1bGxSZXF1ZXN0NDc1NjY1OTg0 | 142 | insert_all(..., alter=True) should work for new columns introduced after the first 100 records | 96218 | closed | 0 | 3 | 2020-08-28T22:22:57Z | 2020-08-30T07:28:23Z | 2020-08-28T22:30:14Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/142 | Closes #139. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/142/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
275939188 | MDU6SXNzdWUyNzU5MzkxODg= | 143 | Mechanism for "suggested visualizations" | 9599 | closed | 0 | 1 | 2017-11-22T04:10:25Z | 2018-07-10T17:48:34Z | 2018-07-10T17:48:34Z | OWNER | Each visualization should have a way of deciding if it might be appropriate for the current view of data. We can then offer a "suggested visualizations" prompt which shows previews. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/143/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
688389933 | MDU6SXNzdWU2ODgzODk5MzM= | 143 | Move to GitHub Actions CI | 9599 | closed | 0 | 1 | 2020-08-28T22:34:11Z | 2020-08-28T22:41:35Z | 2020-08-28T22:41:35Z | OWNER | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/143/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
276091279 | MDU6SXNzdWUyNzYwOTEyNzk= | 144 | apsw as alternative sqlite3 binding (for full text search) | 649467 | closed | 0 | 3 | 2017-11-22T14:40:39Z | 2018-05-28T21:29:42Z | 2018-05-28T21:29:42Z | NONE | Hey there, Have you considered providing apsw support as an alternative to stock python sqlite3? I use apsw because it keeps up with sqlite3 and is straightforward to bring in extensions like FTS5. FTS really accelerates the kind of searching often done by web clients. I may be able to help (it shouldn't be much code), but there are a couple of stylistic questions that come up when supporting an optional package. Also, apsw is tricky in that it doesn't have a pypi package (author says limitations in providing options to setup.py). | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/144/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
688395275 | MDU6SXNzdWU2ODgzOTUyNzU= | 144 | Run some tests against numpy | 9599 | closed | 0 | 2 | 2020-08-28T22:53:00Z | 2020-08-28T22:57:05Z | 2020-08-28T22:57:04Z | OWNER | Accidentally removed in #143: https://github.com/simonw/sqlite-utils/blob/d7d3f962861ef32c5ead8f514c8756f5b6f7c4a0/.travis.yml#L18-L19 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/144/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
276192732 | MDExOlB1bGxSZXF1ZXN0MTU0MjQ2ODE2 | 145 | Fix pytest version conflict | 9599 | closed | 0 | 0 | 2017-11-22T20:15:34Z | 2017-11-22T20:17:54Z | 2017-11-22T20:17:52Z | OWNER | simonw/datasette/pulls/145 | https://travis-ci.org/simonw/datasette/jobs/305929426 pkg_resources.VersionConflict: (pytest 3.2.1 (/home/travis/virtualenv/python3.5.3/lib/python3.5/site-packages), Requirement.parse('pytest==3.2.3')) | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/145/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
688659182 | MDU6SXNzdWU2ODg2NTkxODI= | 145 | Bug when first record contains fewer columns than subsequent records | 96218 | closed | 0 | 2 | 2020-08-30T05:44:44Z | 2020-09-08T23:21:23Z | 2020-09-08T23:21:23Z | CONTRIBUTOR | `insert_all()` selects the maximum batch size based on the number of fields in the first record. If the first record has fewer fields than subsequent records (and `alter=True` is passed), this can result in SQL statements with more than the maximum permitted number of host parameters. This situation is perhaps unlikely to occur, but could happen if the first record had, say, 10 columns, such that `batch_size` (based on `SQLITE_MAX_VARIABLE_NUMBER = 999`) would be 99. If the next 98 rows had 11 columns, the resulting SQL statement for the first batch would have `10 * 1 + 11 * 98 = 1088` host parameters (and subsequent batches, if the data were consistent from thereon out, would have `99 * 11 = 1089`). I suspect that this bug is masked somewhat by the fact that while: > [`SQLITE_MAX_VARIABLE_NUMBER`](https://www.sqlite.org/limits.html#max_variable_number) ... defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0. it is common that it is increased at compile time. Debian-based systems, for example, seem to ship with a version of sqlite compiled with `SQLITE_MAX_VARIABLE_NUMBER` set to 250,000, and I believe this is the case for homebrew installations too. A test for this issue might look like this: ```python def test_columns_not_in_first_record_should_not_cause_batch_to_be_too_large(fresh_db): # sqlite on homebrew and Debian/Ubuntu etc. is typically compiled with # SQLITE_MAX_VARIABLE_NUMBER set to 250,000, so we need to exceed this value to # trigger the error on these systems. THRESHOLD = 250000 extra_columns = 1 + (THRESHOLD - 1) // 99 records = [ {"c0": "first record"}, # one column in first record -> batch_size = 100 # fill out the batch with 99 records with enough columns to exceed THRESHOLD *[ dict([("c{}".format(i), j) for i in range(extra_columns)]) for j in range(99) ] ] try: fresh_db["too_many_columns"].insert_all(records, a… | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/145/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
276455748 | MDU6SXNzdWUyNzY0NTU3NDg= | 146 | datasette publish gcloud | 9599 | closed | 0 | 2 | 2017-11-23T18:55:03Z | 2019-06-24T06:48:20Z | 2019-06-24T06:48:20Z | OWNER | See also #103 It looks like you can start a Google Cloud VM with a "docker container" option - and the Google Cloud Registry is easy to push containers to. So it would be feasible to have `datasette publish gcloud ...` automatically build a container, push it to GCR, then start a new VM instance with it: https://cloud.google.com/container-registry/docs/pushing-and-pulling | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/146/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
688668680 | MDExOlB1bGxSZXF1ZXN0NDc1ODc0NDkz | 146 | Handle case where subsequent records (after first batch) include extra columns | 96218 | closed | 0 | 5 | 2020-08-30T07:13:58Z | 2020-09-08T23:20:37Z | 2020-09-08T23:20:37Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/146 | Addresses #145. I think this should do the job. If it meets with your approval I'll update this PR to include an update to the documentation -- I came across this bug while preparing a PR to update the documentation around `batch_size` in any event. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/146/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
276476670 | MDU6SXNzdWUyNzY0NzY2NzA= | 147 | Tidy up design of the header of the table page | 9599 | closed | 0 | 2919870 | 1 | 2017-11-23T21:52:58Z | 2017-11-24T22:02:46Z | 2017-11-24T22:02:46Z | OWNER | This is a bit messy: <img width="883" alt="fatal-police-shootings-data__fatal-police-shootings-data" src="https://user-images.githubusercontent.com/9599/33189214-9379807c-d055-11e7-96c4-58a70529cef0.png"> Depends on #127 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/147/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
688670158 | MDU6SXNzdWU2ODg2NzAxNTg= | 147 | SQLITE_MAX_VARS maybe hard-coded too low | 96218 | open | 0 | 7 | 2020-08-30T07:26:45Z | 2021-02-15T21:27:55Z | CONTRIBUTOR | I came across this while about to open an issue and PR against the documentation for `batch_size`, which is a bit incomplete. As mentioned in #145, while: > [`SQLITE_MAX_VARIABLE_NUMBER`](https://www.sqlite.org/limits.html#max_variable_number) ... defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0. it is common that it is increased at compile time. Debian-based systems, for example, seem to ship with a version of sqlite compiled with SQLITE_MAX_VARIABLE_NUMBER set to 250,000, and I believe this is the case for homebrew installations too. In working to understand what `batch_size` was actually doing and why, I realized that by setting `SQLITE_MAX_VARS` in `db.py` to match the value my sqlite was compiled with (I'm on Debian), I was able to decrease the time to `insert_all()` my test data set (~128k records across 7 tables) from ~26.5s to ~3.5s. Given that this about .05% of my total dataset, this is time I am keen to save... Unfortunately, it seems that `sqlite3` in the python standard library doesn't expose the `get_limit()` C API (even though `pysqlite` used to), so it's hard to know what value sqlite has been compiled with (note that this could mean, I suppose, that it's less than 999, and even hardcoding `SQLITE_MAX_VARS` to the conservative default might not be adequate. It can also be lowered -- but not raised -- at runtime). The best I could come up with is `echo "" | sqlite3 -cmd ".limits variable_number"` (only available in `sqlite >= 2015-05-07 (3.8.10)`). Obviously this couldn't be relied upon in `sqlite_utils`, but I wonder what your opinion would be about exposing `SQLITE_MAX_VARS` as a user-configurable parameter (with suitable "here be dragons" warnings)? I'm going to go ahead and monkey-patch it for my purposes in any event, but it seems like it might be worth considering. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/147/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
276477888 | MDU6SXNzdWUyNzY0Nzc4ODg= | 148 | Need a != filter | 9599 | closed | 0 | 2919870 | 0 | 2017-11-23T22:05:22Z | 2017-11-23T22:10:02Z | 2017-11-23T22:10:01Z | OWNER | https://datasette-demos.now.sh/sf-trees-ebc2ad9/Street_Tree_List?qCareAssistant=1 shows trees managed by FUF - but how about trees that are NOT managed by FUF? | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/148/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
695276328 | MDU6SXNzdWU2OTUyNzYzMjg= | 148 | More attractive indentation of created FTS table schema | 9599 | closed | 0 | 1 | 2020-09-07T16:49:30Z | 2020-09-07T18:12:50Z | 2020-09-07T18:12:50Z | OWNER | On https://github-to-sqlite.dogsheep.net/github/licenses_fts the create table SQL is displayed as: ```sql CREATE VIRTUAL TABLE [licenses_fts] USING FTS5 ( [name], content=[licenses] ); ``` It would be more aesthetically pleasing if it looked like this: ```sql CREATE VIRTUAL TABLE [licenses_fts] USING FTS5 ( [name], content=[licenses] ); ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/148/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
276704127 | MDU6SXNzdWUyNzY3MDQxMjc= | 149 | Update custom SQL results to match new table view header | 9599 | closed | 0 | 2919870 | 1 | 2017-11-24T22:03:59Z | 2017-11-24T22:42:10Z | 2017-11-24T22:42:09Z | OWNER | Follow-on from #147 - the custom SQL results page should more closely match the design of the table view, which now looks like this: <img width="798" alt="conventional_power_plants_eu__conventional_power_plants_eu__14_rows_where_company____nuon_" src="https://user-images.githubusercontent.com/9599/33224621-4d547f18-d120-11e7-9b33-44ee738a5ea2.png"> | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/149/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
695319258 | MDU6SXNzdWU2OTUzMTkyNTg= | 149 | FTS table with 7 rows has _fts_docsize table with 9,141 rows | 9599 | closed | 0 | 10 | 2020-09-07T18:06:16Z | 2020-09-07T21:16:34Z | 2020-09-07T21:16:34Z | OWNER | I'm seeing a weird issue with some of the SQLite databases that I am using with the FTS5 module. I have a database with a `licenses` table that contains 7 rows: <https://github-to-sqlite.dogsheep.net/github/licenses> The FTS table also has 7 rows: <https://github-to-sqlite.dogsheep.net/github/licenses_fts> Somehow the accompanying `licenses_fts_docsize` shadow table now has 9,141 rows in it! <https://github-to-sqlite.dogsheep.net/github/licenses_fts_docsize> And `licenses_fts_data` has 41 rows - should I expect that to have 7 rows? <https://github-to-sqlite.dogsheep.net/github/licenses_fts_data> I have a hunch that it might be a problem with the triggers. These are the triggers that are updating that FTS table: <https://github-to-sqlite.dogsheep.net/github?sql=select+*+from+sqlite_master+where+type+%3D+%27trigger%27+and+tbl_name+%3D+%27licenses%27> | type | name | tbl_name | rootpage | sql | | --- | --- | --- | --- | --- | | trigger | licenses_ai | licenses | 0 | `CREATE TRIGGER [licenses_ai] AFTER INSERT ON [licenses] BEGIN INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` | | trigger | licenses_ad | licenses | 0 | `CREATE TRIGGER [licenses_ad] AFTER DELETE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); END` | | trigger | licenses_au | licenses | 0 | `CREATE TRIGGER [licenses_au] AFTER UPDATE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` | | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/149/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
276704327 | MDU6SXNzdWUyNzY3MDQzMjc= | 150 | _group_count= feature improvements | 9599 | closed | 0 | 3 | 2017-11-24T22:06:18Z | 2018-05-28T16:41:28Z | 2018-05-28T16:41:28Z | OWNER | - [ ] The "apply filters" form should keep you on the _group_count= page - [ ] Foreign key references should be expand - [ ] Page title should reflect the view you are on | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/150/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
695359607 | MDU6SXNzdWU2OTUzNTk2MDc= | 150 | Feature for tracing SQL queries | 9599 | closed | 0 | 0 | 2020-09-07T19:43:08Z | 2020-09-07T21:57:01Z | 2020-09-07T21:57:01Z | OWNER | Debugging `sqlite-utils` when something weird happens (e.g. #149) can be a bit tricky since it runs a bunch of different SQL statements behind the scenes. An optional "tracing" mechanism for seeing what SQL is being executed would be useful. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/150/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
276718605 | MDU6SXNzdWUyNzY3MTg2MDU= | 151 | Set up a pattern portfolio | 9599 | closed | 0 | 2 | 2017-11-25T02:09:49Z | 2020-07-02T00:13:24Z | 2020-05-03T03:13:16Z | OWNER | https://www.slideshare.net/nataliedowne/practical-maintainable-css/75 This will be a single page that demonstrates all of the different CSS styles and classes available to Datasette. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/151/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed |