- 6551 Views
- 3 replies
- 5 kudos
Resolved! Z-Ordering Timestamp Column
Hi,I've large Delta Table for IoT data for over 10K different sensors with timestamp, sensor name and value columns at 1 second precision.Query pattern is usually random 5-100 sensors at a time. But typically involves specific year/month/day interval...
- 6551 Views
- 3 replies
- 5 kudos
- 5 kudos
@numersoz did you z-order on the timestamp column or on less granular columns, like Year, Month, or Day. timestamp column is very granular (high cardinality) since it also includes hour, minute, second...
- 5 kudos
- 1562 Views
- 1 replies
- 0 kudos
Databricks intellisence adds a second aliad
Hi all, A couple of weeks ago I noticed that whenever I use the intellisense to autocomplete my column names in the sql editor, it adds a second alias. For example if i have a table (Table1) and I alias it in my query (i.e. Table1 as a) if I use the...
- 1562 Views
- 1 replies
- 0 kudos
- 0 kudos
And how do I fix it?
- 0 kudos
- 662 Views
- 0 replies
- 0 kudos
ava.net.SocketTimeoutException - ReadTimeOut
Databricks notebook is connecting ADLS Gen2 using Service principal authentication and the setup is working fine. The notebook is able to read/write files to ADLS Gen2. However, occasionally, we are seeing below error in the production environment:ja...
- 662 Views
- 0 replies
- 0 kudos
- 5502 Views
- 0 replies
- 0 kudos
How to abort/fail a notebook in databricks (and in datafactory)
Hello everyone!I have a pipeline in Data Factory full of notebooks. In one of the notebooks I have the following (inside a function take writes a query to a table):"try:df = spark.sql(query)df.write.format(format).mode(method).saveAsTable(table)excep...
- 5502 Views
- 0 replies
- 0 kudos
- 5160 Views
- 4 replies
- 0 kudos
Why can't I query a table from a cluster, but can query from another cluster in the same workspace
I have two clusters A, B under the same azure databricks workspace. Under cluster A, inside my notebook, I tried to query a table: `SELECT * FROM some_table LIMIT 5`. It shows some permission errors. Under cluster B, if I run the same sql query, it ...
- 5160 Views
- 4 replies
- 0 kudos
- 0 kudos
Check for any spark config or init script differences in the two clusters.
- 0 kudos
- 2725 Views
- 0 replies
- 0 kudos
Azure Databricks: leading zeros in decimal integer literals are not permitted
Hey team,Need your help.I am trying to run the below python code in a data bricks notebook, which is part of parsing an XML file, exploding the element. This works great for all the other elements with no numbers and elements not starting with a zero...
- 2725 Views
- 0 replies
- 0 kudos
- 1788 Views
- 1 replies
- 0 kudos
Streaming tables vs DLT
Have a couple of questions wrt Streaming tables, Kindly help us with this.1)Can we create streaming tables without a DLT pipeline?2)Can we create streaming tables in Databricks SQL?3)What we observe in Streaming tables, it support Kafka and event lo...
- 1788 Views
- 1 replies
- 0 kudos
- 0 kudos
HiPlease refer to the document: https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-create-streaming-table.html#create-streaming-tableI think this should help you answer your questions.
- 0 kudos
- 13059 Views
- 4 replies
- 0 kudos
How can I insert into 2 tables within one database transaction with spark SQL / pyspark?
Hi all,I have a postgres database that contains two tables: A and B.Also, I have 2 delta tables, called C and D. My task is to push the data from A to C and B to D - and if something fails, then leave everything as is.With python it is easy. Set up t...
- 13059 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @thomasthomas We haven't heard from you since the last response from @daniel_sahal , and I was checking back to see if her suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to oth...
- 0 kudos
- 4662 Views
- 3 replies
- 1 kudos
Resolved! How to manage cluster permissions
How to manage different workspaces’ clusters permissions?
- 4662 Views
- 3 replies
- 1 kudos
- 1 kudos
Which {request_object_type} to use for setting permissions for a cluster? "cluster", "clusters", "compute" does not work.
- 1 kudos
- 2191 Views
- 2 replies
- 0 kudos
Unable to login or reset password Community Edition
I'm not able to login databricks community edition from last 5 days, getting Invalid email address or password issue even though its correct credentials. i tried to reset password but unable to reset password also. what is the issue. can you guys pl...
- 2191 Views
- 2 replies
- 0 kudos
- 0 kudos
im getting mails saying that if issue is fixed then mark one as a accepted solution but unfortunately issue is still not yet fixed , can someone look into it and fix issue please ??
- 0 kudos
- 2136 Views
- 1 replies
- 0 kudos
A problem with formating python code to pep8
Hi there,Anyone here experienced any problems with formatiing python code inside databricks notebook cells? I'm constantly having this weird problem, both with very short/simple and complex code, in different notebooks. Sometimes it somehow works, an...
- 2136 Views
- 1 replies
- 0 kudos
- 0 kudos
that is interesting.Can you check if you you use a version > 11.2. If not you have to install black and tokenize-rt.Can you check if you use a custom pyproject.toml file? It might be related to that.(You also need edit permission on the notebook bu...
- 0 kudos
- 2796 Views
- 1 replies
- 0 kudos
Using Libreoffice in Databricks
Hi Community, I'm using Databricks E2, and need to convert pptx files to pdf files.This can be done in either a python or an R notebook using #LibreofficeTo achieve this I'd have to download LibreOffice; I'm not too sure on how to do that. Would I ha...
- 2796 Views
- 1 replies
- 0 kudos
- 0 kudos
I suppose by Libreoffice you mean the sdk, without the frontend?You will have to install the jar as a library on the compute cluster.From that moment on, you can use the classes in your code.If you cannot run the jar from a command line, it might be ...
- 0 kudos
- 1619 Views
- 0 replies
- 0 kudos
Schema Evolution from Kafka Source
Hi,I have a Spark streaming process that reads data from a Kafka topic to Azure DLThis is how I implement the MERGE capability into the delta table.In addition to the same topic, I have another streaming process that simply writes data to DLIn kafka ...
- 1619 Views
- 0 replies
- 0 kudos
- 2171 Views
- 1 replies
- 0 kudos
Clusters do not start - bootstrap timeout
Hello,Whenever I try to start any of already existing clusters, I get Bootstrap timeout error. In the logs, there are following messages:[Bootstrap Event] Can reach databricks-update-oregon.s3.us-west-2.amazonaws.com: [FAILED] [ 257.556698] audit: k...
- 2171 Views
- 1 replies
- 0 kudos
- 0 kudos
The error message indicate the ec2 instance cannot access databricks-update-oregon.s3.us-west-2.amazonaws.com. Do you have s3 endpoint setup or can traffic route to databricks-update-oregon.s3.us-west-2.amazonaws.com ?
- 0 kudos
- 5100 Views
- 1 replies
- 0 kudos
Connect to EXCEL
I have hardtime connecting my existing EXCEL file to source data from DataBricks and need help
- 5100 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Thanks for contacting Databricks Support. We doesn't support direct Excel-Databricks connectivity. However, Databricks can be accessed through ODBC and JDBC interfaces, and we can leverage these with Excel's Power Query functionality for indirect...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 567 | |
| 349 | |
| 287 |
